In this section we study the proposed stabilized immersed isogeometric analysis formulation for a series of test cases. In Section <ref> we consider a binary-fluid Taylor-Couette flow, which we use to benchmark the framework against a conventional boundary-fitted finite element analysis. In Section <ref> an idealized porous medium with a periodic microstructure is considered to demonstrate the modeling capabilities of the proposed framework. Finally, in Section <ref> a porous medium application is considered to demonstrate its ability to handle complex geometries.

4cFluid-flow parameters

Mass densities     $\rho_{1}$     1000     ${\rm kg/m^3}$

$\rho_{2}$     1.3     ${\rm kg/m^3}$

Dynamic viscosities     $\eta_{1}$     $1 \cdot 10^{-3}$     ${\rm Pa\, s}$

$\eta_{2}$     $1.813 \cdot 10^{-5}$     ${\rm Pa\, s}$

Surface tension     $\sigma_{12}$     $72.8\cdot10^{-3}$     N/m

Generalized Navier parameter     $\aGN$     100     ${\rm Pa\,  s/m}$

4cPhase-field parameters

Interface thickness     $\varepsilon$     $0.78125\cdot10^{-6}$     ${\rm m}$

Mobility     $m$     $3.0487\cdot10^{-10}$     ${\rm m\, s^2 /kg}$

4cStabilization parameters

Nitsche penalty     $\beta$     100     –

Skeleton penalty     $\gamma_{\rm skeleton}$     0.01     –

Ghost penalty     $\gamma_{\rm ghost}$     0.01     –

Properties of the Navier-Stokes-Cahn-Hilliard model used for the numerical experiments.

Unless specified otherwise, the parameters of the Navier-Stokes-Cahn-Hilliard model as listed in Table <ref> are used for all test cases. These parameters – which represent a water-air flow – have a strong influence on the mesh and time resolution required to accurately evaluate the model. In particular, the interface thickness, $\varepsilon$, has a strong impact on the element size, and the mobility, $m$, has a strong influence on the time step size. The model parameters are selected such that stable and accurate results can be obtained on uniform background meshes with a moderate number of elements and with a moderate number of time steps. This enables studying the proposed immersed isogeometric analysis framework at an acceptable computational expense using various mesh and time step sizes, as well as studying its robustness with respect to cut-element configurations. The adaptive solution strategy for the NSCH equations presented in Ref. <cit.> can be tailored to our immersed framework (a theoretical basis for this is provided in Ref. <cit.>) to substantially reduce the computational effort associated with the uniform discretization considered here, but this extension is beyond the scope of this work. The test cases are restricted to the two-dimensional case and are implemented in the Python-based (isogeometric) finite element framework Nutils <cit.>. Although there are no fundamental obstacles in extending the work to three dimensions – both in terms of the formulation and in terms of the discretization method – the computational burden associated with three-dimensional simulations would necessitate the usage of an adaptive solution strategy and the implementation in a high-performance computing framework (iterative solvers with preconditioners, scalable parallel implementation, etc.).

The computational domains for the upcoming test cases are constructed by trimming the domain through the octree-tessellation procedure discussed in Section <ref> based on analytical level set functions. The octree depth for all cases is set to 3, meaning that the octree recursion is terminated after 3 element bisections. Fifth-order Gaussian quadrature rules are selected on the integration sub-cells, as they are found to yield a suitable balance between minimizing computational expense and the impact of integration errors on the presented results. It is noted, however, that optimized integration rules can be used to enhance the performance of the framework <cit.>.

For all immersed isogeometric simulations presented below, cubic ($k=3$) B-splines with optimal regularity are employed for all field variables. For the selection of the various stabilization parameters, we follow the empirical rules proposed in Ref. <cit.>. The Nitsche parameter is set to 100 and both the skeleton-penalty parameter and the ghost-penalty parameter are set to 0.01 (see Table <ref>). These parameters are found to provide a good balance between model stability (large enough) and impact on the accuracy of the approximation (small enough), for all considered simulations.

§.§ Taylor-Couette flow

We consider the binary-fluid Taylor-Couette flow between two parallel plates moving in opposite directions, as illustrated in Fig. <ref>, and studied in Refs. <cit.>. The fluid constituents are separated by a vertical interface in the center of the domain. To maintain symmetry, for this benchmark the mass density and viscosity are taken the same for both constituents, i.e., $\rho_1=\rho_2=\rho=1000{\rm kg/m^3}$ and $\eta_1=\eta_2=\eta=1\cdot 10^{-3}{\rm Pa\,s}$. Moreover, the mobility is decreased to $m=3.0487\cdot10^{-11}{\rm m\,s^2/kg}$. The speed of the plate is increased gradually over the first second per $u_{\rm wall}=\frac{1}{2}\big(1-\cos(\pi\,t)\big) 10\,{\rm m/s}$, after which it is kept constant at $u_{\rm wall}=10{\rm m/s}$ until a steady state solution is obtained. In the initial state, the vertical diffuse interface is prescribed using an analytical approximation in accordance with the selected model
parameters. On the left and right (far field) boundaries, a linear velocity profile  is imposed, where $u_{\rm slip}=(1+\frac{2\eta}{\aGN H })^{-1}u_{\rm wall}$. Such a profile corresponds to the far-field (pure species) Taylor-Couette steady state solution compatible with the generalized Navier boundary condition. The phase field is set to +1 on the left boundary, and to -1 on the right boundary.

Illustration of the domain and boundary conditions for the binary-fluid Taylor-Couette benchmark case.

[The boundary-fitted mesh used to compute the finite element analysis reference result.][The background mesh for the immersed isogeometric analysis simulations. Rotated by an angle $\theta$.]
Illustration of the computational meshes used for the binary-fluid Taylor-Couette benchmark.

A boundary-fitted mesh can straightforwardly be constructed on the rectangular domain considered here (Fig. <ref>), making this test case suitable for benchmarking against a classical finite element discretization (Section <ref>). To study the proposed immersed isogeometric analysis method, the rectangular domain is immersed in a rotated ambient domain, as illustrated in Fig. <ref>. By varying the rotation of the ambient domain, $\theta$, the robustness with respect to cut-element configurations can be studied (Section <ref>).

To provide intuition for the studies presented in the remainder of this section, in Fig. <ref> we illustrate the evolution of the phase field (left column) and velocity field (right column) obtained using an ambient domain mesh with element size $h=0.3125{\rm \mu\rm m}$, which is rotated by $\theta=\frac{\pi}{8}{\rm rad}$ in counter clock-wise direction. The top row shows the solution in the initial state, where the diffuse interface is vertical and the velocity is zero. At $t=0.5{\rm s}$ (second row), the plates are moving, which results in the interface being dragged along with the plates, resulting in a (mildly) curved interface. The generalized Navier boundary condition is observable through the mismatch between the speed of the fluid at the plate boundaries (approximately $1.7{\rm m/s}$ away from the interface) and that of the plates ($u_{\rm wall}=5{\rm m/s}$). As time progresses, and the plate velocity is increased up to the maximum speed at $t=1.0{\rm s}$ (third row), the curvature of the interface increases, until a steady-state solution is obtained (bottom row). Note that the slip-velocity-based Reynolds number ${\rm Re} = 33$ is sufficiently small to allow for a steady state solution, and that convective effects are small enough as not to require the use of convection-stabilization. In this steady state, the maximum rotation of the interface relative to its (vertical) initial position is $0.23{\rm rad}$, and the tangential stresses due to capillary effects induce sufficient slip to equilibrate the meniscus-to-wall contact point. Away from this contact point, the classical wedge-shape Huh-Scriven velocity profiles are retrieved <cit.>.

Time evolution of the phase field $\varphi$ (left) and the velocity field $\u$ (right) for the Taylor-Couette test case. The presented results are obtained using immersed isogeometric analysis with a $h=0.3125μm$ ambient domain mesh rotated by $θ= π/8rad$.}
\label{fig:tcsnapshots}
\end{figure}

\subsubsection{Benchmarking against boundary-fitted finite elements}
\label{sec:tcbenchmark}
\noindent
To benchmark the proposed immersed isogeometric formulation, we compare the discretization described in \cref{sec:immersed,sec:WeakFormulation} to a
boundary-fitted simulation, for which all essential boundary conditions can be imposed strongly: since the plates are aligned with the horizontal direction, the normal component of the velocity can be constrained by removal of the basis functions corresponding to the vertical velocity component. Taylor-Hood $C^0$-continuous finite elements with cubic velocity functions and quadratic pressure functions are considered to obtain a stable velocity-pressure discretization. The phase field and chemical potential field are discretized using cubic finite elements. In this boundary-fitted finite element setting, no additional stabilization of the weak form \eqref{eq:weak} is required.

In Fig.~\ref{fig:tcbenchmarkphase}, the steady-state phase field obtained using traditional boundary-fitted finite element analysis (FEA) (left column) and the proposed immersed isogeometric analysis (IGA) (right column) are compared. In the top row, an element size of $h=0.625μm$ is used, resulting in a mesh with $80 ×16=1280$ elements for the boundary-fitted case. The bottom row presents the results for an element size of $h=0.3125μm$, resulting in a boundary-fitted mesh with $160 ×32=5120$ elements. For both meshes, the obtained phase-field solutions for the boundary-fitted and immersed simulations are virtually indistinguishable, demonstrating the consistency of the stabilized immersed IGA formulation. The maximum interface rotation angle for the immersed case is within a few percent of the FEA reference result. This difference can be attributed to the fact that the far-field in/out-flow conditions are imposed at a finite distance from the interface. The error associated with this inconsistent application of the far-field condition differs between the FEA case and IGA case on account of the size and rotation of the ambient domain (\emph{i.e.}, the boundaries are located in a different position). Due to the optimal-regularity B-splines used in the isogeometric analysis, the number of degrees of freedom associated with these simulations is substantially (approximately 6 times) smaller than that for the FEA case.

Fig.~\ref{fig:tcbenchmarkdiv} compares the divergence of the velocity field between the traditional FEA case and the immersed IGA, considering the same meshes as for the phase-field solutions in Fig.~\ref{fig:tcbenchmarkphase}. Although, like the phase field, the velocity fields (not shown here) are virtually indistinguishable between the two methods, notable differences can be observed in the divergence of the velocity field. Since the mass conservation balance \eqref{eq:weak_mass} is solved weakly for both the FEA case and the immersed IGA case, discretization errors result in a non-zero divergence for both analyses, in particular in the vicinity of the interface. For both FEA and immersed IGA the error in the mass conservation decreases under mesh refinement, although the errors observed using traditional FEA are substantially smaller than for the immersed IGA case. We attribute this increased error in the velocity-divergence for the immersed IGA case to the reduced number of degrees of freedom in combination with the skeleton-penalty term required for stabilization. Although this reduction does not noticeably affect the phase field and the velocity field, it does impact the local conservation properties. Furthermore, it is observed that while the errors in the divergence are essentially restricted to the interface in the boundary-fitted case, in the immersed case errors are also observed along the immersed boundaries. This is due to the penalized and conflicting nature of the divergence-free condition and the convective impermeability condition in the cut elements. In our numerical experiments we have not experienced any negative consequences due to this phenomenon, but it should be recognized as a potential source of instabilities.

\begin{figure}
\begin{tabular}{cc}
\subfloat[Coarse mesh ($h=0.625{\rm \mu m}$) boundary fitted FEA with 51,778 dofs.]{\includegraphics[width=0.49\textwidth,trim={0 0 0 2.6cm},clip]{figures/TaylorCouette_BF/phi_TC_bf_nx80_t1_25.png}}
\subfloat[Coarse mesh ($h=0.625{\rm \mu m}$) immersed IGA with 9,384 dofs.]{\includegraphics[width=0.49\textwidth,trim={0 0 0 2.6cm},clip]{figures/TaylorCouette_immersed/phi_TC_im_nx80_t1_25.png}}
\\
\subfloat[Fine mesh ($h=0.3125{\rm \mu m}$) boundary fitted FEA with 205,954 dofs.]{\includegraphics[width=0.49\textwidth,trim={0 0 0 2.6cm},clip]{figures/TaylorCouette_BF/phi_TC_bf_nx160_t1_25.png}}
\subfloat[Fine mesh ($h=0.3125{\rm \mu m}$) immersed IGA with 32,582 dofs.]{\includegraphics[width=0.49\textwidth,trim={0 0 0 2.6cm},clip]{figures/TaylorCouette_immersed/phi_TC_im_nx160_t1_25_cb.png}}
\\
\end{tabular}
\caption{Comparison between a boundary fitted FEA and an immersed IGA of the steady state phase-field solution, $\varphi(t=\infty)$, for a coarse and a fine mesh.
\end{figure}

\begin{figure}
\begin{tabular}{cc}
\subfloat[Coarse mesh ($h=0.625{\rm \mu m}$) boundary fitted FEA with 51,778 dofs.]{\includegraphics[width=0.49\textwidth]{figures/TaylorCouette_BF/divu_TC_bf_nx80_t1_25.png}}
\subfloat[Coarse mesh ($h=0.625{\rm \mu m}$) immersed IGA with 9,384 dofs.]{\includegraphics[width=0.49\textwidth]{figures/TaylorCouette_immersed/divu_TC_im_nx80_t1_25.png}}
\\
\subfloat[Fine mesh ($h=0.3125{\rm \mu m}$) boundary fitted FEA with 205,954 dofs.]{\includegraphics[width=0.49\textwidth]{figures/TaylorCouette_BF/divu_TC_bf_nx160_t1_25.png}}
\subfloat[Fine mesh ($h=0.3125{\rm \mu m}$) immersed IGA with 32,582 dofs.]{\includegraphics[width=0.49\textwidth]{figures/TaylorCouette_immersed/divu_TC_im_nx160_t1_25.png}}
\\
\end{tabular}
\caption{Comparison between a boundary fitted FEA and an immersed IGA of the steady state velocity-divergence solution, $\nabla\cdot\u(t=\infty)$, for a coarse and a fine mesh. Note that the color bars have different ranges.}
\label{fig:tcbenchmarkdiv}
\end{figure}

\subsubsection{Robustness with respect to cut-element configurations}
\label{sec:tcorientation}
\noindent
To study the robustness of the proposed formulation in terms of how elements are cut, various rotations of the ambient domain mesh are considered. Fig.~\ref{fig:tcorientation} shows the phase field (left column) and velocity field (right column) for three different orientations of the ambient mesh. The considered mesh size is $h=0.3125μm$. The top row concerns a very small rotation angle of 0.001rad. This mesh rotation leads to very thin sliver cuts along the boundaries. The presented results are virtually indistinguishable from the reference solution discussed above. The absence of instabilities associated with the Nitsche boundary condition and ill-conditioning problems \cite{deprenter2023stability} conveys that the ghost-penalty stabilization is effective. The middle and bottom rows of Fig.~\ref{fig:tcorientation} pertain to the case of a $π/8$rad and $π/4$rad rotation of the ambient mesh, respectively. Also for these orientations, the obtained solutions are indistinguishable from the reference solution, despite the different cut-element shapes caused by the different rotations.

\begin{figure}
\begin{tabular}{cc}
\centering
\subfloat[$\theta=0.001$rad]{\includegraphics[width=0.36\textwidth]{figures/TaylorCouette_immersed/phi_TC_im_nx160_0_t1_25.png}}
\subfloat[ $\theta=0.001$rad]{\includegraphics[width=0.36\textwidth]{figures/TaylorCouette_immersed/velocity_TC_im_nx160_0_t1_25.png}}
\\
\subfloat[$\theta=\frac{\pi}{8}$rad]{\includegraphics[width=0.41\textwidth]{figures/time_series/phi_TC_im_nx160_t1_25.png}}
\subfloat[$\theta=\frac{\pi}{8}$rad]{\includegraphics[width=0.41\textwidth]{figures/time_series/velocity_TC_im_nx160_t1_25.png}}
\\
\subfloat[$\theta=\frac{\pi}{4}$rad]{\includegraphics[width=0.49\textwidth]{figures/TaylorCouette_immersed/phi_TC_im_nx160_pi4_t1_25.png}}
\subfloat[$\theta=\frac{\pi}{4}$rad]{\includegraphics[width=0.49\textwidth]{figures/TaylorCouette_immersed/velocity_TC_im_nx160_pi4_t1_25.png}}
\\
\end{tabular}
\caption{Steady-state phase-field, $\varphi(t=\infty)$, and velocity field, $\boldsymbol{u}(t=\infty)$, solutions computed with the immersed IGA framework on ambient domain meshes with a mesh size of $h=0.3125\mu\rm m$ and various mesh rotations $\theta$.}
\label{fig:tcorientation}
\end{figure}

\subsection{Lattice of circular inclusions}
\label{sec:inclusion}
\noindent
To demonstrate the capability of the immersed isogeometric analysis framework to handle arbitrary geometries, we consider a water-air binary-fluid system (\cref{tab:properties}) flowing through a lattice of circular inclusions (Fig.~\ref{fig:latticedomain}). The simulated unit cell has a width of $40μm$ and a height of $20μm$. The circular inclusions have a radius of $10μm$ and an offset of $40μm$ in both the horizontal and vertical direction. In the immersed isogeometric analysis framework, this problem requires a rectangular ambient domain corresponding to the bounding box of the unit cell. Construction of a boundary-fitted isogeometric analysis mesh is possible, but would require a multi-patch description.

\begin{figure}
\centering
\includegraphics[width=0.99\textwidth]{figures/Domain_sketches/Periodic_bubble_domain.pdf}
\caption{Illustration of the domain and boundary conditions for the binary-fluid flow through a lattice of circular inclusions.}
\label{fig:latticedomain}
\end{figure}

The flow through the lattice is forced by means of a constant, horizontally oriented, body force $F_b=1·10^9 N/m^3$. In line with the lattice interpretation of this problem, we consider periodic boundary conditions for the problem on the left and right boundaries, meaning that the field variables attain identical values on corresponding points on the inflow and outflow boundaries. In our simulations, this periodic behavior is enforced by constructing an ambient domain mesh with periodic B-splines in the horizontal direction. Note that this construction is trivial on account of the rectangular ambient domain. On the top and bottom boundaries of the domain, symmetry boundary conditions are considered, meaning that the vertical velocity vanishes, as do the horizontal
traction and the normal gradients of the phase field and chemical potential.

We discretize the ambient domain with elements of size $h=0.25μm$. After trimming, this results in a discrete problem with 43,085 degrees of freedom. The initial time step size is set to $Δt=0.5μs$. With interface velocities as high as approximately $70m/s$, the use of time step adaptation is required. In the presented results this is implemented by dividing the time step by two when the nonlinear problem fails to converge, and restoring to the original time step after 8 converged time steps using the smaller step size; see also Ref.~\cite{Demont:2022dk}.
In the initial state we consider the two phases to be distributed as vertically oriented lamellae, with the interfaces positioned at the shortest distances between the obstacles, as illustrated in Fig.~\ref{fig:latticedomain}. The horizontal body force drives these lamellae past the obstacles, requiring them to break up and eventually to realign with the flow direction.

\begin{figure}
\begin{tabular}{cc}
\centering
\subfloat[$t=4.25\mu \rm s$]{\includegraphics[width=0.49\textwidth]{figures/Immersed_bubbles/phi_bp_t4_25.png}\label{fig:latticefirstcyclea}}
\subfloat[$t=4.25\mu \rm s$]{\includegraphics[width=0.49\textwidth]{figures/Immersed_bubbles/velocity_bp_t4_25.png}\label{fig:latticefirstcycleb}}
\\
\subfloat[$t=5.625\mu \rm s$]{\includegraphics[width=0.49\textwidth]{figures/Immersed_bubbles/phi_bp_t5_625.png}\label{fig:latticefirstcyclec}}
\subfloat[$t=5.625\mu \rm s$]{\includegraphics[width=0.49\textwidth]{figures/Immersed_bubbles/velocity_bp_t5_625.png}\label{fig:latticefirstcycled}}
\\
\subfloat[$t=6.25\mu \rm s$]{\includegraphics[width=0.49\textwidth]{figures/Immersed_bubbles/phi_bp_t6_25.png}\label{fig:latticefirstcyclee}}
\subfloat[$t=6.25\mu \rm s$]{\includegraphics[width=0.49\textwidth]{figures/Immersed_bubbles/velocity_bp_t6_25.png}\label{fig:latticefirstcyclef}}
\\
\end{tabular}
\caption{Time evolution of the phase field $\varphi$ (left) and velocity field $\u$ (right), showing the motion of the lamellae past the obstacles.}
\end{figure}

\begin{figure}\ContinuedFloat
\begin{tabular}{cc}
\centering
\subfloat[$t=13.4375\mu \rm s$]{\includegraphics[width=0.49\textwidth]{figures/Immersed_bubbles/phi_bp_t13_4375.png}\label{fig:latticefirstcycleg}}
\subfloat[$t=13.4375\mu \rm s$]{\includegraphics[width=0.49\textwidth]{figures/Immersed_bubbles/velocity_bp_t13_4375.png}\label{fig:latticefirstcycleh}}
\\
\subfloat[$t=15.75\mu\rm s$]{\includegraphics[width=0.49\textwidth]{figures/Immersed_bubbles/phi_bp_t15_75.png}\label{fig:latticefirstcyclei}}
\subfloat[$t=15.75\mu\rm s$]{\includegraphics[width=0.49\textwidth]{figures/Immersed_bubbles/velocity_bp_t15_75.png}\label{fig:latticefirstcyclej}}
\\
\subfloat[$t=16.875\mu\rm s$]{\includegraphics[width=0.49\textwidth]{figures/Immersed_bubbles/phi_bp_t16_875.png}\label{fig:latticefirstcyclek}}
\subfloat[$t=16.875\mu\rm s$]{\includegraphics[width=0.49\textwidth]{figures/Immersed_bubbles/velocity_bp_t16_875.png}\label{fig:latticefirstcyclel}}
\\
\end{tabular}
\caption{Time evolution of the phase field $\varphi$ (left) and velocity field $\u$ (right), showing the motion of the lamellae past the obstacles.}
\label{fig:latticefirstcycle}
\end{figure}

\begin{figure}
\begin{tabular}{cc}
\centering
\subfloat[$t=\infty$]{\includegraphics[width=0.49\textwidth]{figures/Immersed_bubbles/phi_bp_tsteady.png}\label{fig:latticefirstcyclet25.5mus}}
\subfloat[$t=\infty$]{\includegraphics[width=0.49\textwidth]{figures/Immersed_bubbles/velocity_bp_tsteady.png}}
\\
\end{tabular}
\caption{The phase field $\varphi$ (left) and velocity field $\u$ (right), showing the steady state of the flow in a porous medium with circular inclusions.}
\label{fig:latticesteady}
\end{figure}

Fig.~\ref{fig:latticefirstcycle} illustrates the transient behavior of the lamellae passing through the unit cell.
Initially, the water lamella (corresponding to $\varphi = 1$, shown in dark blue) enters the domain from the left. As the flow velocity is stalled by the obstacles, the fluids move fastest near the center of the free space between the inclusions. This causes the interfaces to curve, as observed already in Fig.~\ref{fig:latticefirstcyclea}. In this figure, the air-water interface approaches the upper-right obstacle. Due to the low velocities close to the obstacle boundary, a thin layer of water remains, eventually resulting in the trapped water droplet shown in Fig.~\ref{fig:latticefirstcyclec}. Due to the Ostwald ripening effect, this satellite droplet is quickly re-absorbed by the inflowing water lamella, see Fig.~\ref{fig:latticefirstcyclee}. Fig.~\ref{fig:latticefirstcycleg} shows the phase field after the lamellae have moved through two complete cycles of the periodic unit-cell. At this time instance, the water lamella coalesces with the detaching tail of the upstream lamella. After the isolation and re-absorption of a few more droplets, which may be observed in \cref{fig:latticefirstcycleg,fig:latticefirstcyclei}, the fully re-segregated and realigned state shown in Fig.~\ref{fig:latticefirstcyclek} is realized after approximately 5 cycles. The steady state, illustrated in Fig.~\ref{fig:latticesteady}, is reached shortly after.
In their new horizontal orientation, no further topological changes to the lamellae are induced by the obstacles.

\subsection{Porous medium}
\label{sec:porousmedium}
\noindent
To illustrate the capabilities of the immersed isogeometric analysis framework, we study the evolution of a water-air binary-fluid system propagating through the example porous medium depicted in Fig.~\ref{fig:porousdomain}. The width and height of the domain are $200\mu\rm m$ and $50\mu\rm m$, respectively. On the left boundary of the domain, the velocity is prescribed by a horizontal flow with a parabolic profile which satisfies the generalized Navier condition. The maximum velocity of the inflow profile is set to $5 {\rm m/s}$. On the inflow boundary, the phase field is constrained to $\varphi=1$, representing the (dark blue) water phase. On the top boundary, symmetry conditions are imposed, similar to the lattice problem discussed above. Outflow boundary conditions are considered on the bottom and right boundaries, meaning that the traction is set to zero. We make use of a mesh size of $h=0.625\mu\rm m$, leading to 185,618 degrees-of-freedom systems of equations, and a time step size of $\Delta t=0.05{\rm \mu s}$.

\begin{figure}[!b]
\centering
\includegraphics[width=0.95\textwidth]{figures/Domain_sketches/Porous_domain.pdf}
\caption{Illustration of the domains and inflow conditions for the binary flow through an example porous medium.\\}
\label{fig:porousdomain}
\end{figure}

\begin{figure}
\begin{tabular}{cc}
\centering
\subfloat[$t=0\mu \rm s$]{\includegraphics[width=0.49\textwidth]{figures/Porous_medium/phi_por_t0.png}\label{fig:poroussnapshotsa}}
\subfloat[$t=0\mu \rm s$]{\includegraphics[width=0.49\textwidth]{figures/Porous_medium/velocity_por_t0.png}}
\\
\subfloat[$t=9\mu \rm s$]{\includegraphics[width=0.49\textwidth]{figures/Porous_medium/phi_por_t9_0.png}\label{fig:poroussnapshotsc}}
\subfloat[$t=9\mu \rm s$]{\includegraphics[width=0.49\textwidth]{figures/Porous_medium/velocity_por_t9_0.png}}
\\
\subfloat[$t=10\mu \rm s$]{\includegraphics[width=0.49\textwidth]{figures/Porous_medium/phi_por_t10_0.png}\label{fig:poroussnapshotse}}
\subfloat[$t=10\mu \rm s$]{\includegraphics[width=0.49\textwidth]{figures/Porous_medium/velocity_por_t10_0.png}}
\\
\end{tabular}
\caption{Time evolution of the phase field $\varphi$ (left) and velocity field $\u$ (right) for the porous medium test case.}
\end{figure}

\begin{figure}\ContinuedFloat
\begin{tabular}{cc}
\centering
\subfloat[$t=17.0\mu \rm s$]{\includegraphics[width=0.49\textwidth]{figures/Porous_medium/phi_por_t17_0.png}\label{fig:poroussnapshotsg}}
\subfloat[$t=17.0\mu \rm s$]{\includegraphics[width=0.49\textwidth]{figures/Porous_medium/velocity_por_t17_0.png}}
\\
\subfloat[$t=25.9\mu \rm s$]{\includegraphics[width=0.49\textwidth]{figures/Porous_medium/phi_por_t25_9.png}\label{fig:poroussnapshotsi}}
\subfloat[$t=25.9\mu \rm s$]{\includegraphics[width=0.49\textwidth]{figures/Porous_medium/velocity_por_t25_9.png}}
\\
\subfloat[$t=27.15\mu \rm s$]{\includegraphics[width=0.49\textwidth]{figures/Porous_medium/phi_por_t27_15.png}\label{fig:poroussnapshotsk}}
\subfloat[$t=27.15\mu \rm s$]{\includegraphics[width=0.49\textwidth]{figures/Porous_medium/velocity_por_t27_15.png}}
\\
\end{tabular}
\caption{Time evolution of the phase field $\varphi$ (left) and velocity field $\u$ (right) for the porous medium test case.}
\end{figure}

\begin{figure}\ContinuedFloat
\begin{tabular}{cc}
\centering
\subfloat[$t=31.5\mu \rm s$]{\includegraphics[width=0.49\textwidth]{figures/Porous_medium/phi_por_t31_5.png}\label{fig:poroussnapshotsm}}
\subfloat[$t=31.5\mu \rm s$]{\includegraphics[width=0.49\textwidth]{figures/Porous_medium/velocity_por_t31_5.png}}
\\
\subfloat[$t=40.0\mu \rm s$]{\includegraphics[width=0.49\textwidth]{figures/Porous_medium/phi_por_t40_0.png}\label{fig:poroussnapshotso}}
\subfloat[$t=40.0\mu \rm s$]{\includegraphics[width=0.49\textwidth]{figures/Porous_medium/velocity_por_t40_0.png}}
\\
\subfloat[$t=43.9\mu \rm s$]{\includegraphics[width=0.49\textwidth]{figures/Porous_medium/phi_por_t43_9.png}\label{fig:poroussnapshotsq}}
\subfloat[$t=43.9\mu \rm s$]{\includegraphics[width=0.49\textwidth]{figures/Porous_medium/velocity_por_t43_9.png}}
\\
\end{tabular}
\caption{Time evolution of the phase field $\varphi$ (left) and velocity field $\u$ (right) for the porous medium test case.}
\label{fig:poroussnapshots}
\end{figure}
Fig.~\ref{fig:poroussnapshotsa} shows the initial condition, where the diffuse interface is positioned at the inflow boundary. At $t=9{\rm \mu s}$ the water-front, $\varphi=1$, has moved inside the domain and meets the strong curvature of the bottom boundary of the solid domain. This stagnates the motion of the interface along that boundary. At $t=10{\rm \mu s}$, in \cref{fig:poroussnapshotse}, the inflow overcomes this stagnation, resulting in high velocities and rapid change of the interface shape. In \cref{fig:poroussnapshotsg}, the inflow phase branches between the leftmost outflow channel and the top channel, following the motion of the fluid. At $t=25.9{\rm \mu s}$ two trapped air bubbles (corresponding to $\varphi=-1$, shown in light blue) emerge at the top symmetry boundary. The small air bubble is quickly re-absorbed by the larger bubble, as illustrated in the progression snapshots of \cref{fig:poroussnapshotsi,fig:poroussnapshotsk,fig:poroussnapshotsm}. Since the velocity in this region is very small, this isolated region becomes stable in time. The high radius-of-curvature of the solid wall at the top effectively pins the water-front from \cref{fig:poroussnapshotsi} until \cref{fig:poroussnapshotsq}, where the second outflow is reached, creating a second entrapment of air in the porous domain.
\end{document}


* [176] H. Dai, X. Fang, J. Jia, Consensus-based distributed fixed-time optimization for a class of resource allocation problems, Journal of the Franklin Institute 359 (18) (2022) 11135–11154.
  * [177] Y. Song, J. Cao, L. Rutkowski, A fixed-time distributed optimization algorithm based on event-triggered strategy, IEEE Transactions on Network Science and Engineering 9 (3) (2021) 1154–1162.
  * [178] G. Chen, Z. Guo, Initialization-free distributed fixed-time convergent algorithms for optimal resource allocation, IEEE Transactions on Systems, Man, and Cybernetics: Systems 52 (2) (2020) 845–854.
  * [179] X. Shi, L. Xu, T. Yang, Z. Lin, X. Wang, Distributed fixed-time resource allocation algorithm for the general linear multi-agent systems, IEEE Transactions on Circuits and Systems II: Express Briefs 69 (6) (2022) 2867–2871.
  * [180] Z. Li, Z. Ding, Distributed multiobjective optimization for network resource allocation of multiagent systems, IEEE Transactions on Cybernetics 51 (12) (2020) 5800–5810.
  * [181] Z. Guo, G. Chen, Distributed dynamic event-triggered and practical predefined-time resource allocation in cyber–physical systems, Automatica 142 (2022) 110390.
  * [182] X. Gong, Y. Cui, J. Shen, J. Xiong, T. Huang, Distributed optimization in prescribed-time: Theory and experiment, IEEE Transactions on Network Science and Engineering 9 (2) (2022) 564–576.
  * [183] W. Lin, Y. Wang, C. Li, X. Yu, Predefined-time optimization for distributed resource allocation, Journal of the Franklin Institute 357 (16) (2020) 11323–11348.
  * [184] X. Wang, Y. Hong, X. Sun, K. Liu, Distributed optimization for resource allocation problems under large delays, IEEE Transactions on Industrial Electronics 66 (12) (2019) 9448–9457.
  * [185] S. S. Kia, Distributed optimal in-network resource allocation algorithm design via a control theoretic approach, Systems & Control Letters 107 (2017) 49–57.
  * [186] W. Lin, Y. Wang, C. Li, X. Yu, Distributed resource allocation via accelerated saddle point dynamics, IEEE/CAA Journal of Automatica Sinica 8 (9) (2021) 1588–1599.
  * [187] M. Doostmohammadian, U. A. Khan, A. Aghasi, Distributed constraint-coupled optimization over unreliable networks, in: 10th RSI International Conference on Robotics and Mechatronics (ICRoM), IEEE, 2022, pp. 371–376.
  * [188] L. Li, D. Ho, J. Lu, A consensus recovery approach to nonlinear multi-agent system under node failure, Information Sciences 367 (2016) 975–989.
  * [189] S. Sundaram, B. Gharesifard, Secure local filtering algorithms for distributed optimization, in: IEEE 55th Conference on Decision and Control (CDC), IEEE, 2016, pp. 1871–1876.
  * [190] A. Falsone, K. Margellos, J. Zizzo, M. Prandini, S. Garatti, On the sensitivity of linear resource sharing problems to the arrival of new agents, IEEE Transactions on Automatic Control (2022).
  * [191] W. Fu, Q. Ma, J. Qin, Y. Kang, Resilient consensus-based distributed optimization under deception attacks, International Journal of Robust and Nonlinear Control 31 (6) (2021) 1803–1816.
  * [192] G. Shao, R. Wang, X. Wang, K. Liu, Distributed algorithm for resource allocation problems under persistent attacks, Journal of the Franklin Institute 357 (10) (2020) 6241–6256.
  * [193] A. Sardana, R. Joshi, An auto-responsive honeypot architecture for dynamic resource allocation and qos adaptation in ddos attacked networks, Computer Communications 32 (12) (2009) 1384–1399.
  * [194] B. Turan, C. A. Uribe, H. Wai, M. Alizadeh, Resilient primal–dual optimization algorithms for distributed resource allocation, IEEE Transactions on Control of Network Systems 8 (1) (2020) 282–294.
  * [195] C. Xu, Q. Liu, T. Huang, Resilient penalty function method for distributed constrained optimization under byzantine attack, Information Sciences 596 (2022) 362–379.
  * [196] R. Wang, Y. Liu, Q. Ling, Byzantine-resilient resource allocation over decentralized networks, IEEE Transactions on Signal Processing 70 (2022) 4711–4726.
  * [197] C. Xu, Q. Liu, A resilient distributed optimization algorithm based on consensus of multi-agent system against two attack scenarios, Journal of the Franklin Institute (2022).
  * [198] M. Yemini, A. Nedić, S. Gil, A. J. Goldsmith, Resilience to malicious activity in distributed optimization for cyberphysical systems, in: IEEE 61st Conference on Decision and Control (CDC), IEEE, 2022, pp. 4185–4192.
  * [199] O. Beaude, P. Benchimol, S. Gaubert, P. Jacquot, N. Oudjane, A privacy-preserving method to optimize distributed resource allocation, SIAM Journal on Optimization 30 (3) (2020) 2303–2336.
  * [200] S. Angel, S. Kannan, Z. Ratliff, Private resource allocators and their applications, in: IEEE Symposium on Security and Privacy, IEEE, 2020, pp. 372–391.
  * [201] W. Lin, Y. Wang, C. Li, X. Yu, Distributed resource allocation: an indirect dual ascent method with an exponential convergence rate, Nonlinear Dynamics 102 (2020) 1685–1699.
  * [202] G. Chen, Z. Li, Distributed optimal resource allocation over strongly connected digraphs: A surplus-based approach, Automatica 125 (2021) 109459.
  * [203] P. Jacquot, O. Beaude, P. Benchimol, S. Gaubert, N. Oudjane, A privacy-preserving disaggregation algorithm for non-intrusive management of flexible energy, in: IEEE 58th Conference on Decision and Control (CDC), IEEE, 2019, pp. 890–896.
  * [204] A. Nedic, A. Ozdaglar, Distributed subgradient methods for multi-agent optimization, IEEE Transactions on Automatic Control 54 (1) (2009) 48–61.
  * [205] M. Doostmohammadian, Distributed energy resource management: All-time resource-demand feasibility, delay-tolerance, nonlinearity, and beyond, IEEE Control Systems Letters (2023).
  * [206] M. Doostmohammadian, A. Aghasi, M. Vrakopoulou, H. R. Rabiee, U. A. Khan, T. Charalambous, Distributed delay-tolerant strategies for equality-constraint sum-preserving resource allocation, Systems & Control Letters (2023).
  * [207] A. Seuret, D. Dimarogonas, K. H. Johansson, Consensus under communication delays, in: 47th IEEE Conference on Decision and Control, IEEE, 2008, pp. 4922–4927.
  * [208] C. N. Hadjicostis, T. Charalambous, Average consensus in the presence of delays in directed graph topologies, IEEE Transactions on Automatic Control 59 (3) (2014) 763–768.
  * [209] M. Doostmohammadian, U. Khan, M. Pirani, T. Charalambous, Consensus-based distributed estimation in the presence of heterogeneous, time-invariant delays, IEEE Control Systems Letters 6 (2021) 1598–1603.
  * [210] L. Schenato, F. Fiorentin, Average timesynch: A consensus-based protocol for clock synchronization in wireless sensor networks, Automatica 47 (9) (2011) 1878–1886.
  * [211] L. Schenato, G. Gamba, A distributed consensus protocol for clock synchronization in wireless sensor network, in: 46th ieee conference on decision and control, IEEE, 2007, pp. 2289–2294.
  * [212] G. Chen, Z. Zhao, Delay effects on consensus-based distributed economic dispatch algorithm in microgrid, IEEE Transactions on Power Systems 33 (1) (2017) 602–612.
  * [213] F. Fagnani, S. Zampieri, Average consensus with packet drop communication, SIAM Journal on Control and Optimization 48 (1) (2009) 102–133.
  * [214] N. Vaidya, C. Hadjicostis, A. Domínguez-García, Robust average consensus over packet dropping links: Analysis via coefficients of ergodicity, in: IEEE 51st IEEE Conference on Decision and Control (CDC), IEEE, 2012, pp. 2761–2766.
  * [215] Z. Liu, X. You, H. Yang, L. Zhao, Leader-following consensus of heterogeneous multi-agent systems with packet dropout, International Journal of Control, Automation and Systems 13 (2015) 1067–1075.
  * [216] C. Hadjicostis, N. Vaidya, A. Domínguez-García, Robust distributed average consensus via exchange of running sums, IEEE Transactions on Automatic Control 61 (6) (2015) 1492–1507.
  * [217] X. Xia, A. M. Elaiw, Optimal dynamic economic dispatch of generation: A review, Electric power systems research 80 (8) (2010) 975–986.
  * [218] B. Chowdhury, S. Rahman, A review of recent advances in economic dispatch, IEEE Transactions on Power Systems 5 (4) (1990) 1248–1259.
  * [219] W. Elsayed, E. El-Saadany, A fully decentralized approach for solving the economic dispatch problem, IEEE Transactions on power systems 30 (4) (2014) 2179–2189.
  * [220] M. Firouzbahrami, A. Nobakhti, Finite-time distributed economic dispatch over network systems with coupled local costs, IEEE Control Systems Letters 7 (2022) 325–330.
  * [221] A. Cherukuri, S. Martínez, J. Cortés, Distributed, anytime optimization in power-generator networks for economic dispatch, in: American Control Conference, IEEE, 2014, pp. 172–177.
  * [222] H. Pourbabak, J. Luo, T. Chen, W. Su, A novel consensus-based distributed algorithm for economic dispatch based on local estimation of power mismatch, IEEE Transactions on Smart Grid 9 (6) (2017) 5930–5942.
  * [223] A. I. Rikos, J. Nylöf, S. Gracy, K. H. Johansson, Distributed optimal allocation with quantized communication and privacy-preserving guarantees, IFAC-PapersOnLine 55 (41) (2022) 64–70.
  * [224] A. Grammenos, T. Charalambous, E. Kalyvianaki, CPU scheduling in data centers using asynchronous finite-time distributed coordination mechanisms, IEEE Transactions on Network Science and Engineering (2023).
  * [225] E. Kalyvianaki, T. Charalambous, S. Hand, Self-adaptive and self-configured CPU resource provisioning for virtualized servers using kalman filters, in: Proceedings of the 6th International Conference on Autonomic Computing, 2009, pp. 117–126.
  * [226] E. Makridis, K. Deliparaschos, E. Kalyvianaki, A. Zolotas, T. Charalambous, Robust dynamic CPU resource provisioning in virtualized servers, IEEE Transactions on Services Computing 15 (2) (2020) 956–969.
  * [227] X. Chen, L. Yang, Z. Chen, G. Min, X. Zheng, C. Rong, Resource allocation with workload-time windows for cloud-based software services: a deep reinforcement learning approach, IEEE Transactions on Cloud Computing (2022).
  * [228] G. Hattab, S. Ucar, T. Higuchi, O. Altintas, F. Dressler, D. Cabric, Optimized assignment of computational tasks in vehicular micro clouds, in: 2nd International Workshop on Edge Systems, Analytics and Networking, 2019, pp. 1–6.
  * [229] Z. Mann, Allocation of virtual machines in cloud data centers–a survey of problem models and optimization algorithms, ACM Computing Surveys (CSUR) 48 (1) (2015) 1–34.
  * [230] N. Li, C. Zhao, L. Chen, Connecting automatic generation control and economic dispatch from an optimization view, IEEE Transactions on Control of Network Systems 3 (3) (2015) 254–264.
  * [231] P. Kumar, D. Kothari, Recent philosophies of automatic generation control strategies in power systems, IEEE transactions on power systems 20 (1) (2005) 346–357.
  * [232] K. Ullah, A. Basit, Z. Ullah, S. Aslam, H. Herodotou, Automatic generation control strategies in conventional and modern power systems: A comprehensive overview, Energies 14 (9) (2021) 2376.
  * [233] H. Bevrani, T. Hiyama, Intelligent automatic generation control, CRC press, 2017\.
  * [234] Q. Zhou, M. Shahidehpour, A. Paaso, S. Bahramirad, A. Alabdulwahab, A. Abusorrah, Distributed control and communication strategies in networked microgrids, IEEE Communications Surveys & Tutorials 22 (4) (2020) 2586–2633.
  * [235] A. J. Wood, B. F. Wollenberg, G. B. Sheblé, Power generation, operation, and control, John Wiley & Sons, 2013.
  * [236] S. Sun, Q. Yang, W. Yan, Optimal temporal-spatial PEV charging scheduling in active power distribution networks, Protection and Control of Modern Power Systems 2 (2017) 1–10.
  * [237] M. Kisacikoglu, F. Erden, N. Erdogan, Distributed control of PEV charging based on energy demand forecast, IEEE Transactions on Industrial Informatics 14 (1) (2017) 332–341.
  * [238] J. Mukherjee, A. Gupta, Distributed charge scheduling of plug-in electric vehicles using inter-aggregator collaboration, IEEE Transactions on Smart Grid 8 (1) (2016) 331–341.
  * [239] M. Li, J. Gao, N. Chen, L. Zhao, X. Shen, Decentralized PEV power allocation with power distribution and transportation constraints, IEEE Journal on Selected Areas in Communications 38 (1) (2019) 229–243.
  * [240] A. Abdalrahman, W. Zhuang, A survey on PEV charging infrastructure: Impact assessment and planning, Energies 10 (10) (2017) 1650.
  * [241] D. Dallinger, M. Wietschel, Grid integration of intermittent renewable energy sources using price-responsive plug-in electric vehicles, Renewable and Sustainable Energy Reviews 16 (5) (2012) 3370–3382.
  * [242] A. Amin, W. Tareen, M. Usman, H. Ali, I. Bari, B. Horan, S. Mekhilef, M. Asif, S. Ahmed, A. Mahmood, A review of optimal charging strategy for electric vehicles under dynamic pricing schemes in the distribution charging network, Sustainability 12 (23) (2020) 10160.
  * [243] M. Esmaili, M. Rajabi, Optimal charging of plug-in electric vehicles observing power grid constraints, IET Generation, Transmission & Distribution 8 (4) (2014) 583–590.
  * [244] A. Falsone, K. Margellos, S. Garatti, M. Prandini, Dual decomposition for multi-agent distributed optimization with coupling constraints, Automatica 84 (2017) 149–158.
  * [245] R. Vujanic, P. M. Esfahani, P. J. Goulart, S. Mariéthoz, M. Morari, A decomposition method for large scale milps, with performance guarantees and a power system application, Automatica 67 (2016) 144–156.
  * [246] L. Deori, K. Margellos, M. Prandini, On decentralized convex optimization in a multi-agent setting with separable constraints and its application to optimal charging of electric vehicles, in: IEEE 55th Conference on Decision and Control (CDC), IEEE, 2016, pp. 6044–6049.
  * [247] D. Ioan, I. Prodan, S. Olaru, F. Stoican, S. Niculescu, Mixed-integer programming in motion planning, Annual Reviews in Control 51 (2021) 65–87.
  * [248] X. Li, X. Yi, L. Xie, Distributed online optimization for multi-agent networks with coupled inequality constraints, IEEE Transactions on Automatic Control 66 (8) (2020) 3575–3591.
  * [249] M. Chiang, S. H. Low, A. R. Calderbank, J. C. Doyle, Layering as optimization decomposition: A mathematical theory of network architectures, Proceedings of the IEEE 95 (1) (2007) 255–312.
  * [250] D. P. Palomar, M. Chiang, A tutorial on decomposition methods for network utility maximization, IEEE Journal on Selected Areas in Communications 24 (8) (2006) 1439–1451.
  * [251] X. Lin, N. B. Shroff, R. Srikant, A tutorial on cross-layer optimization in wireless networks, IEEE Journal on Selected areas in Communications 24 (8) (2006) 1452–1463.
  * [252] R. Srikant, L. Ying, Communication networks: an optimization, control, and stochastic networks perspective, Cambridge University Press, 2013.
  * [253] N. Hasan, W. Ejaz, N. Ejaz, H. S. Kim, A. Anpalagan, M. Jo, Network selection and channel allocation for spectrum sharing in 5g heterogeneous networks, IEEE Access 4 (2016) 980–992.
  * [254] G. I. Tsiropoulos, O. A. Dobre, M. H. Ahmed, K. E. Baddour, Radio resource allocation techniques for efficient spectrum access in cognitive radio networks, IEEE Communications Surveys & Tutorials 18 (1) (2014) 824–847.
  * [255] M. Kraning, E. Chu, J. Lavaei, S. Boyd, Dynamic network energy management via proximal message passing, Foundations and Trends® in Optimization 1 (2) (2013) 73–126.
  * [256] D. Trossen, G. Parisis, Designing and realizing an information-centric internet, IEEE Communications Magazine 50 (7) (2012) 60–67.
  * [257] O. Ercetin, L. Tassiulas, Market-based resource allocation for content delivery in the internet, IEEE Transactions on Computers 52 (12) (2003) 1573–1585.
  * [258] S. Guo, J. Liu, Y. Yang, B. Xiao, Z. Li, Energy-efficient dynamic computation offloading and cooperative task scheduling in mobile cloud computing, IEEE Transactions on Mobile Computing 18 (2) (2018) 319–333.
  * [259] D. Ergu, G. Kou, Y. Peng, Y. Shi, Y. Shi, The analytic hierarchy process: task scheduling and resource allocation in cloud computing environment, The Journal of Supercomputing 64 (2013) 835–848.
  * [260] M. Chiang, T. Zhang, Fog and iot: An overview of research opportunities, IEEE Internet of things journal 3 (6) (2016) 854–864.
  * [261] F. P. Kelly, A. K. Maulloo, D. K. H. Tan, Rate control for communication networks: shadow prices, proportional fairness and stability, Journal of the Operational Research society 49 (1998) 237–252.
  * [262] C. Enyioha, A. Jadbabaie, V. Preciado, G. Pappas, Distributed resource allocation for control of spreading processes, in: 2015 European Control Conference (ECC), IEEE, 2015, pp. 2216–2221.
  * [263] C. Nowzari, V. M. Preciado, G. J. Pappas, Analysis and control of epidemics: A survey of spreading processes on complex networks, IEEE Control Systems Magazine 36 (1) (2016) 26–46.
  * [264] D. Peteiro-Barral, B. Guijarro-Berdiñas, A survey of methods for distributed machine learning, Progress in Artificial Intelligence 2 (2013) 1–11.
  * [265] Y. Kumar, S. Kaul, Y. Hu, Machine learning for energy-resource allocation, workflow scheduling and live migration in cloud computing: State-of-the-art survey, Sustainable Computing: Informatics and Systems 36 (2022) 100780.
  * [266] A. Mohajer, M. Barari, H. Zarrabi, Big data based self-optimization networking: A novel approach beyond cognition, Intelligent Automation & Soft Computing (2017) 1–7.


# FogROS2-SGC: A ROS2 Cloud Robotics Platform
for Secure Global Connectivity

Kaiyuan Chen1, Ryan Hoque1, Karthik Dharmarajan1, Edith LLontop1, Simeon
Adebola1,
Jeffrey Ichnowski2, John Kubiatowicz1, and Ken Goldberg1,3 1Department of
Electrical Engineering and Computer Science2Carnegie Mellon University,
Pittsburgh, PA, USA3Department of Industrial Engineering and Operations
Research1,3University of California, Berkeley, CA, USA<EMAIL_ADDRESS>
###### Abstract

The Robot Operating System (ROS2) is the most widely used software platform
for building robotics applications. FogROS2 extends ROS2 to allow robots to
access cloud computing on demand. However, ROS2 and FogROS2 assume that all
robots are locally connected and that each robot has full access and control
of the other robots. With applications like distributed multi-robot systems,
remote robot control, and mobile robots, robotics increasingly involves the
global Internet and complex trust management. Existing approaches for
connecting disjoint ROS2 networks lack key features such as security,
compatibility, efficiency, and ease of use. We introduce FogROS2-SGC, an
extension of FogROS2 that can effectively connect robot systems across
different physical locations, networks, and Data Distribution Services (DDS).
With globally unique and location-independent identifiers, FogROS2-SGC
securely and efficiently routes data between robotics components around the
globe. FogROS2-SGC is agnostic to the ROS2 distribution and configuration, is
compatible with non-ROS2 software, and seamlessly extends existing ROS2
applications without any code modification. Experiments suggest FogROS2-SGC is
19$\times$ faster than rosbridge (a ROS2 package with comparable features, but
lacking security). We also apply FogROS2-SGC to 4 robots and compute nodes
that are 3600 km apart. Videos and code are available on the project website
https://sites.google.com/view/fogros2-sgc.

##  I Introduction

As robots are increasingly deployed worldwide, they require mechanisms to
efficiently, reliably, and securely communicate with other robots, sensors,
computers, and the cloud. The applications are broad, from mobile robots with
changing IP addresses due to traveling through different networks, to a fleet
of globally distributed robots learning collaboratively. Cloud and fog
robotics [1] empower robots and automation systems to harness off-board
resources in cloud-based computers. In prior work, we introduced FogROS2 [2],
now an official part of the ROS2 ecosystem [3], to enable robots to execute
modern compute and memory-intensive algorithms using on-demand hardware
resources on the edge and cloud. However, robots connecting to the cloud, a
nearby computer on a different network, or a robot halfway around the world
introduce additional challenges: (1) Making robots accessible to other robots
on the public internet may leave them vulnerable to unauthorized connections
and data breaches. (2) The heterogeneity of interconnected devices,
communication protocols, and configurations causes incompatibilities that
hinder integration and operation. (3) The changing network topology of mobile
robots and Unmanned Aerial Vehicles (UAVs) challenges their ability to stay
connected. To illustrate some of these challenges (Fig. 1), consider:

Figure 1: FogROS2-SGC enables Secure Global Connectivity for robots, allowing
robots to communicate with other robots, computers, and the cloud through a
familiar ROS2 interface. With FogROS2-SGC, (A) drones navigating large
construction sites can seamlessly communicate, even when their IP addresses
are constantly changing due to switching Wi-Fi and cellular networks; (B)
shipping and stocking robots from different corporations can securely share
only the required topics necessary to facilitate the transfer of goods at a
warehouse; and (C) globally distributed robots can participate in fleet
learning. In experiments, we demonstrate FogROS2-SGC on Fleet-DAgger [4], a
fleet learning algorithm, with 4 robot arms operating simultaneously in
different locations.

(A) Security and inspection drones: Drones navigate a construction site and
stream data to a central station that updates a dynamically-changing SLAM map
[5]. As drones fly through different cellular and Wi-Fi networks, their IP
addresses change, but they should remain securely connected.

(B) Coordinating heterogeneous mobile robots in a warehouse: Robots belonging
to different companies (e.g., shipping vs. warehouse) and of different makes
and models hand off items between container and warehouse. Each robot has
unique software packages and versions (e.g., operating systems and network
protocols) and must communicate, but only a few selected topics are necessary
for the handoff.

(C) Distributed fleet learning: Robot arms at different locations pool their
data and collectively update a shared control policy. Robots unable to make
progress can fall back on remote human teleoperators, using algorithms such as
Fleet-DAgger [4].

To address these challenges, we present FogROS2-SGC (Secure Global
Connectivity), an extension of FogROS2 that securely and reliably connects
robots across different software components, network protocols, and physical
locations. FogROS2-SGC enables disjoint ROS2 networks to connect to ROS2 topic
interfaces named with globally-unique and location-independent identifiers.
The robots using FogROS2-SGC can roam freely while staying connected because
the identifiers are constant. The identifiers are 256-bit strings that are
secure by construction—a brute-force attack would have to find a match among
$10^{77}$ possibilities (a value close to the number of protons in the
observable universe111The Eddington Number [6] ($N_{\mathrm{Edd}}$) is
currently estimated to be $10^{80}$.). FogROS2-SGC adopts a security-first
routing design, where only authenticated parties can connect to the robot and
establish secure communication. In contrast to prior work such as SROS2 [7]
and FogROS2 [2], FogROS2-SGC does not require merging distributed ROS2
networks, allowing robots to keep their ROS2 networks private and expose
public topics only if explicitly configured. Providing fine-grain isolation
and access control reduces the attack surface and enhances scalability.

FogROS2-SGC seamlessly integrates with ROS2 applications without code
modifications via an SGC proxy. Its implementation and security policy
configuration are agnostic to ROS2 distributions and their network transport
middleware vendors. FogROS2-SGC is also compatible with non-ROS2 programs that
interact with ROS2 components and can provide secure global connectivity to
non-cloud servers and computers. Furthermore, since memory copy and
synchronization operations are expensive for memory-constrained robots, the
implementation of FogROS2-SGC processes can route data without performing
unnecessary copies (also known as “zero copy”).

Experiments suggest that FogROS2-SGC reduces the network latency of a cloud-
based grasp planning application by 9.42$\times$ compared to unsecured rosduct
[8]-rosbridge [9]. We also deploy FogROS2-SGC to simultaneously control a
fleet of 4 robot arms in different physical locations with compute off-loaded
to a server 3600 km away.

This paper makes the following contributions:  (1) FogROS2-SGC, an extension
of FogROS2 that connects disjoint ROS2 networks by assigning public ROS2
topics with globally-unique and location-independent identifiers.  (2) Method
for secure and efficient routing with FogROS2-SGC.  (3) A Rust implementation
of FogROS2-SGC that uses zero-copy message processing and asynchronous network
operations for robots with memory and compute constraints.  (4) Evaluation of
FogROS2-SGC on cloud robotics applications (vSLAM, grasp planning, motion
planning, simultaneous fleet control) demonstrating up to 9.42$\times$ latency
reduction and enhanced usability.

Figure 2: Comparison of FogROS2-SGC with other distributed ROS2 systems. In
this table, we compare the feature support of different distributed ROS2
systems with the features in Section III. Some features can be supported but
require non-trivial effort beyond changing the configuration. For example,
both the routing service in rti_connext and discovery server in FastDDS/SROS2
support global connectivity but require manually modifying routing rules or
setting up a point-to-point VPN when a new node joins [10]. Rosbridge and
FogROS2 support only unidirectional global and resilient connectivity (marked
with *), meaning that one side of the communication must have a fixed IP. In
contrast, the identifier-based routing of FogROS2-SGC allows either side to
have a dynamic IP address.

##  II Related Work

James Kaufner introduced the term ’Cloud Robotics’ in 2010 [1]. Cloud and fog
computing have been applied to robotic tasks such as grasp planning (Tian et
al. [11], Kehoe et al. [12], and Li et al. [13]), parallelized Monte-Carlo
grasp perturbation sampling (Kehoe et al. [14, 15, 16]), and motion planning
(Lam et al. [17]). Chen et al. [18] and Ichnowski et al. [2]. propose
frameworks for offloading computation to resources on the edge or cloud, while
Ichnowski et al. [19] and Anand et al. [20] present systems that leverage
serverless computing [21]. Modern computing paradigms have enabled new
applications such as multi-robot interactive fleet learning (Swamy et al.
[22], Hoque et al. [4]) and remote sharing of robot systems (Tanwani et al.
[23], Bauer et al. [24]).

Remote interactions between robots and the cloud raise security,
compatibility, and connectivity challenges for robots. Virtual Private
Networks (VPNs) are the most common approach for establishing secure
communication between robots and the cloud for both ROS and ROS2 (e.g., Lim et
al. [25]). Establishing a VPN link between a robot and the cloud is a complex
process [26]. FogROS [18] and FogROS2 [2] automate the certificate generation
and VPN setup. SROS2 [7] is an alternative approach to securing ROS2
communication that enforces access control of ROS2 topics. However, it
requires DDS-dependent discovery mechanisms to ensure connectivity. Discovery
mechanisms for DDS (such as the discovery server for FastDDS [27] and the RTI
routing service for RTI Connext [28]) are vendor-specific and not compatible
with other DDS implementations. Zenoh for ROS2 [29] is integrated with
CycloneDDS to enhance peer-to-peer connectivity, but it is not compatible with
other DDS implementations. ROS Remote [30] by Pereira et al. and MSA [31] by
Xu et al. propose alternative protocols to unify cloud-robot communication.
However, alternative protocols require modifications to ROS applications and
are not compatible with ROS2. Finally, rosbridge [9] proposed by Crick et al.
is widely adopted by both ROS1 and ROS2 to allow non-ROS software to interact
with ROS2 nodes. It can also be used to bridge two non-compatible and remote
ROS applications when used in conjunction with rosduct [8]. However, rosduct
and rosbridge have significantly high message latency when the message size is
large (e.g., images). A summary of how FogROS2-SGC differs from related work
can be found in Fig. 2.

##  III Ten FogROS2-SGC Features

FogROS2-SGC addresses the Secure Global Connectivity (SGC) problem of securely
and reliably connecting globally distributed robots, sensors, computers, and
the cloud by extending FogROS2. We enumerate 10 new features to differentiate
from related libraries and alternative approaches.

#### Globally identifiable addresses

FogROS2-SGC enables a scalable number of ROS2 networks to publish a subset of
ROS2 topics to other disjoint ROS2 networks around the globe. In scenario (C)
from Fig. 1, the robot arms are located at different geographic locations with
different local ROS2 networks. FogROS2-SGC allows remote human teleoperators
to operate the robot arms as if the arms are connected to local networks.
FogROS2-SGC also allows disjoint robots to publish to the same local ROS2
topics with globally unique and identifiable addresses.

#### Transparency to ROS2 applications

ROS2 modularizes a robotics application into nodes, and connects the nodes
into a graph. Nodes communicate with each other through a publish-subscribe
(pub/sub) system, where publisher nodes send messages to topics, and nodes
subscribed to these topics receive these messages. FogROS2-SGC adheres to the
abstractions and interfaces of ROS2. ROS2 applications interact with remote
nodes as if they are nodes on the same robot or subnetwork.

#### Communication security

FogROS2-SGC guarantees that no unauthorized attacker can eavesdrop or tamper
with ROS2 messages. Authorization is identified by user-configured
cryptographic keys. In all three scenarios from Section I, the communication
goes through wide-area network with untrusted infrastructure. FogROS2-SGC
prevents attackers from accessing any content in ROS2 messages and
differentiates authentic robots from spoofing attackers.

#### Global anonymity

Authorized participants can deterministically derive global identifiable
addresses with ROS2 topic information and cryptographic secrets. Attackers
cannot reverse any information used to recover addresses or topics.
FogROS2-SGC guards against attackers knowing part of the ROS2 topic
information deducing the global address. For example, the attackers who know
the topic name and type information cannot guess the address, because they
miss the author information and security credentials of ROS2 node.

#### ROS2 network isolation and topic-level access control

FogROS2-SGC connects robots without merging distributed ROS2 networks. Every
robot can have an arbitrary number of private ROS2 topics and only public
interfaces are shared with other authorized ROS2 networks. Other ROS2 nodes
interact with these public interfaces just as they interact with a local ROS2
topic. This protects the privacy of the robot and prevents unintended messages
from being shared with other disjoint networks of the system. For example, in
scenario (B), a delivering robot from one company and receiving robot from
another may have some proprietary topics that are kept private from each
other. FogROS2-SGC isolates the topics private to the robots.

#### Global connectivity

Some robots are connected to subnetworks that are not directly accessible from
the outside. For example, robots in scenario (C) are in local area networks
behind Network Address Translation (NAT). NAT allows multiple robots to share
the same IP, but the translation is dynamic and ROS2 nodes outside cannot
directly access the robots. FogROS2-SGC can connect ROS2 nodes that are behind
firewalls and NAT.

#### Seamless and resilient connectivity to network dynamism

FogROS2-SGC adapts to the dynamic network behaviors of drones and mobile
robots. FogROS2-SGC does not rely on static IP addresses to identify the
robots because such addresses are usually bound to a physical location. Adding
or reconnecting to robots should not restart the ROS2 node entirely, as this
causes service interruptions and failures.

#### DDS-agnostic compatibility

ROS2 adopts the Data Distribution Service (DDS) as its underlying network
transport middleware to marshal, unmarshal, and exchange messages. ROS2
supports different DDS implementations, such as CycloneDDS [32], FastDDS [27],
and RTI Connext [28]. However, a warehouse in scenario (B) may have robots
running different versions of ROS2 and DDS. FogROS2-SGC is DDS-agnostic by
leveraging ROS2 abstractions and not using any DDS-specific interfaces.

#### Compatibility with non-ROS2 software

FogROS2-SGC allows non-ROS2 software to interact with ROS2 nodes. This
principle is inspired by rosbridge [9]. Besides common transport protocols,
while ROS2 officially supports C++ and Python, FogROS2-SGC allows programs to
use gRPC [33], the most widely used Remote Procedure Call (RPC) framework that
can run in heterogeneous environments and major popular programming languages
(such as Go, Java, Javascript, PHP, Rust), to control the robots.

#### Efficient message processing and routing

ROS2 messages are buffered in memory to be processed by FogROS2-SGC. Because
robots often have memory and compute resource constraints, FogROS2-SGC is
memory-efficient by reducing unnecessary message copying and memory
synchronization. Since FogROS2-SGC requires frequent exchange of packets to
and from the network, network operations (such as send and recv) are not on
the critical path of message processing.

##  IV FogROS2-SGC Design

Figure 3: System overview of FogROS2-SGC’s architecture showing a connection
between a robot camera stream (on the ROS2 topic /image) and the cloud. The
FogROS2-SGC assigns the ROS2 topic /image an anonymous, globally-unique and
location-independent 256-bit identifier [75e...0fc] (truncated for brevity).
The messages between identifiers are securely routed with SGC router.

See (Fig. 3) for system architecture. FogROS2-SGC sends messages via a
globally unique identifier (Sec. IV-A). This identifier is unique to a robot
and topic pair; thus, it can be used for sending and receiving messages
regardless of robot location or network address (Sec. IV-B). The identifier is
secure, and communication is encrypted, meaning only authorized robots and
nodes can access messages from its referenced topic (Sec. IV-C). To implement
routing based on the identifier, FogROS2-SGC consists of two main software
components—(1) a router (Sec. IV-B and IV-C), responsible for securely routing
messages between other routers and nodes, and (2) a proxy (Sec IV-D), that
converts between ROS2 messages and the secure routers. As robots can be
compute- and memory-constrained, FogROS2-SGC provides an efficient
implementation (Sec. IV-E).

###  IV-A Global Addressability

Maintaining a globally unique identifier enables the identification of a
specific robotic component across subnetworks. FogROS2-SGC uses ROS2 topics as
the minimal granularity for the global identifier because a topic is an
interface to ROS2 nodes, and a ROS2 node can publish or subscribe to multiple
ROS2 topics at the same time. For example, a ROS2 vSLAM node in openVSLAM [5]
has four ROS2 topics for camera information, video streaming, output
localization, and mapping information. These ROS2 topics expose standardized
interfaces with fixed message types. Users can limit the exposure of the ROS2
network by allowing only parts of the interface to be public. Partitioning
public and private interfaces also enhances privacy and isolation, prevents
unintended message exchanges, and reduces communication overhead.

The identifier is designed to be unique, deterministic, and location-
independent. To avoid name collisions, every identifier has 256 binary bits,
leading to $2^{256}$ possible identifiers. Instead of letting users decide,
all identifiers are cryptographically derived from the metadata of the ROS2
topics by an identifier manager in SGC proxy. The SGC proxy collects metadata
such as the ROS2 node’s name, author, maintainer, interface, and description
from standard ROS2 interface and user configuration file. The metadata also
has a unique string in case the user needs to deploy the same topic at
different locations. Every topic has an associated security certificate in
X.509 [34] to verify the identity of those who want to publish or subscribe to
the network. All the metadata is serialized and converted into a 256-bit
string using SHA-256 [35], a widely used cryptographic hashing algorithm that
maps arbitrary lengths of text to almost-unique 256-bit binary strings.

Security Analysis: The hashed string is suitable for use as the globally
unique identifier for the followingreasons: (1) Deterministic: The hash is
deterministic so that every party holding the same metadata can derive the
same hash value and thus the same global identifier. (2) One-way: SHA-256 is a
one-way function, so the attacker cannot deduce or reverse the original
metadata from the 256-bit identifier. (3) Avalanche effect: A small change to
the original metadata leads to a new hash value that appears unrelated to the
original hash value. (4) Large namespace: There are $2^{256}$ possible
identifiers and it has been proved to be computationally intractable to find
two messages with the same hash. Verification of these guarantees can be found
in Appel [36].

###  IV-B Location-Independent Routing

Although having all identifiers in the same globally-flat namespace protects
the privacy of the node’s identity information and physical location, the
identifiers do not carry any routing information. Flipping a bit in the
identifier may lead to a drastic change in its physical location, or from
existent to nonexistent. Therefore, securely routing messages between flat
identifiers is a challenging problem. To solve this problem, FogROS2-SGC
consolidates and extends the Global Data Plane (GDP) [37], a peer-to-peer
network that routes messages between location-independent identifiers. The
routers are set up by the user and peer-wise connected into a routing graph;
robots do not need to know other robots’ addresses as long as there is a
connected routing path. The routers can be any machine that has network and
general compute capabilities, such as an edge computer or a cloud server.
Every router stores the mapping between the identifiers and the corresponding
routing information of the identifiers in Routing Information Base (RIB).

A joining robot or router broadcasts an advertisement packet that announces
the existence of the identifier and the routing information to the robot. The
packet format is aligned with other FogROS2-SGC packets in Fig. 5. Other
routers store the routing information in RIB and broadcast the advertisement
packet. Routing is achieved by looking up the destination routing information
in the RIB and forwarding to that destination.

Fig. 4 illustrates a step-by-step example of a publishing and subscribing
/camera topic with FogROS2-SGC. The figure assumes that all the connections
between routers are established. This can be achieved through configuration or
dynamic node discovery [38]. The steps are:  (1) The robot SGC proxy P1
generates an advertisement message for the ROS2 topic /camera and sends it to
Router 1.  (2) After verifying the advertisement message, Router 1 records the
advertisement in its RIB and forwards the topic information to Router 2. There
can be multiple routers between Router 1 and Router 2.  (3) Cloud SGC proxy P2
requests to subscribe to /camera, and the subscribe request is sent to Router
2.  (4) The subscribe request from Router 2 is routed to Router 1 by checking
the source information at Router 2’s RIB. After verifying the request, Router
1’s RIB records P2 as the data sink.  (5) The subscribe request from Router 1
is routed to the robot by checking the source information at Router 1’s RIB.
If the destination is not found, the router broadcasts a query to other
routers.  (6) The robot’s ROS2 publisher sends a ROS2 message to the proxy.
The proxy forwards it to Router 1, Router 1 forwards to Router 2, and Router 2
to the cloud subscriber. At each hop, the messages are forwarded from source
to sink.

Figure 4: An illustration of how a routing connection is established between
robot and cloud. The steps are further described in Section IV-B. (1,2)
Advertisement generation and publish. (3,4,5) Subscribe request. (6) Data
routing.

###  IV-C Secure Communication

Figure 5: An illustration of the cryptographic tools used by SGC proxy to
protect a ROS2 string message. The FogROS2-SGC Packet on the right is the
message that is routed by FogROS2-SGC. The payload is encrypted to protect the
confidentiality of the original ROS2 message. The encrypted data is hashed so
that the receiver can verify the message is intact. The hash is signed with
the sender’s key so that the receiver can verify that the message comes from
an authentic and authorized sender.

The security of the communication is achieved by using a secure network
protocol between routers. We use Datagram Transport Layer Security (DTLS) [35]
to provide communications privacy. The DTLS protocol provides secure and
authenticated communication on User Datagram Protocol (UDP) and includes a
built-in mechanism for dealing with lost or out-of-order packets. DTLS on UDP
is well suited for latency-critical robotics communications systems, due to
its lightweight nature and low overhead compared to transmission Control
Protocl(TCP). The cryptographic algorithms used to secure ROS2 packet
generation process can be found in Fig. 5. The message has the following
security guarantees: Confidentiality: The ROS2 messages are encrypted with AES
Encryption [39] to ensure that only parties with the correct cryptographic key
can decrypt the original ROS2 message data. Integrity: The encrypted message
is hashed by SHA-256 [36] so the receiver or third-party auditor can easily
verify that the message is intact and no other attacker has tampered with the
message. Authenticity: The hashed message is signed by the RSASSA-PSS [40]
algorithm so that receivers can verify that the message is sent from an
authorized sender.

To tailor the security with the communication patterns of robotics
applications, FogROS2-SGC allows flexible peering with other routers or end
points. One may choose to use a dedicated DTLS connection per ROS2 topic,
which is ideal for large message payload and frequent communication (e.g.,
video streaming). One may also choose to use a shared DTLS tunnel, where
multiple ROS2 topics share the same DTLS connection. Sharing the same
connection reduces the cost of secure connection management and message
processing, which is good for small message payloads and less frequent
communication.

###  IV-D Transparent and Compatible SGC proxy

SGC proxy is the interface between FogROS2-SGC and the ROS2 network. In order
to allow seamless integration with any unmodified ROS 2 application code and
mainstream DDS vendors, SGC proxy converts between ROS2 communication and
FogROS2-SGC communication bidirectionally. The user first identifies ROS2
topics that they wish to publish or subscribe through a configuration file.
The proxy launches a local ROS2 publisher or subscriber for the corresponding
topic. New messages from the local ROS2 network are actively subscribed to by
the proxy, and sent to the FogROS2-SGC network. Once the verified subscribers
receive the messages, they convert them to standard ROS messages and publish
to their local ROS2 network.

To allow non-ROS2 programs to communicate with ROS2 nodes, SGC proxy converts
ROS2 messages to a unified JSON-based message format in transit. As a result,
FogROS2-SGC can be extended to a variety of protocols such as TCP, UDP, DTLS,
TLS, and gRPC. Note, however, that some of the protocols need special handling
to be aligned with FogROS2-SGC. For example, gRPC requires the IP addresses of
both robot and cloud for bi-directional message passing.

###  IV-E Compute and Memory-Efficient SGC router

FogROS2-SGC can be deployed on low-power robots under memory and compute
constaints, so an efficient implementation of routing algorithm in Section
IV-B is crucial to the overall performance of the system. Fig. 6 shows an
architecture of SGC router. An idiomatic workflow of the router implementation
is to (1) receive data from ROS2/network, (2) decide which network connection
to forward, and (3) forward data to ROS2/network. Because FogROS2-SGC needs to
be extensible to heterogeneous network protocols, the router needs to maintain
many simultaneous network connections, ranging from ROS2’s publish/subscribe
protocol to general network protocol such as DTLS.

Because low-power robots run under memory constaints, memory copying
operations and synchronization operations (such as mutex) are expensive. SGC
router is implemented in Rust [41] to eliminate memory copying operations and
the need for synchronization. Rust is a programming language that features a
single ownership model: every data object has a single owner, and passing the
data is moving the ownership from one variable to another. As a result, it
prevents race conditions and reduces data copying by enforcing the passing of
data objects by references instead of values.

Robots with few CPU cores usually have low network performance, because
network operations are usually blocking, where the entire packet processing
halts and waits for the network operations to finish. FogROS2-SGC improves CPU
utilization by leveraging asynchronous Rust interfaces [42]. Asynchronous
interfaces are non-blocking, removing the network operations out of the
critical path of message processing.

Figure 6: SGC router architecture. (Orange) Subscribe to a local ROS2 network
and publish to FogROS2-SGC routing network. (Magenta) Receive from FogROS2-SGC
routing network and publish to local ROS2 network. (Black) Intermediate SGC
router that facilitates message routing. SGC router asynchronously reads,
writes, and manages all the network connections. All the message passing
(arrows) is zero-copy and does not require movement of actual messages.

##  V Evaluation

We evaluate FogROS2-SGC on system benchmarks to show how it performs over
alternative designs and on robotics benchmarks to show how robotics
applications benefit from FogROS2-SGC. We also demonstrate FogROS2-SGC on four
physical robot arms running Fleet-DAgger [4], a multi-robot learning
application. In Sections V-A and V-B, we use an Intel NUC with an Intel®
Pentium® Silver J5005 CPU @ 1.50 GHz with a 5 Mbps network connection to act
as the robot. The robot is connected with a Standard DS3 v2 cloud instance (4
vCPUs, 14 GiB memory) on Microsoft Azure. The robot is located at California
(west coast of US), and the cloud server is located at Virginia (east coast of
US).

###  V-A System Benchmarks

We evaluate the performance of FogROS2-SGC’s message processing latency and
throughput against other distributed ROS2 systems. Messages are sent in binary
with type sensor_msgs/CompressedImage and response with string type
std_msgs/String. We compare against the following baselines (1) VPN: We use
Wireguard VPN [43], which is the same VPN as FogROS2 [19] (2) Rosbridge:
Rosbridge is the most commonly used websocket proxy that allows non-ROS code
to interact with ROS code. We use Rosbridge in combination with Rosduct in the
same way as in FogROS [18]. (3) Capsule: We use Capsule, a software switch
inspired by Netbricks [44], to emulate the design of FogROS2-SGC. We also
implement rosduct [8] in ROS2 that converts between ROS2 and network traffic.
The detailed description and implementation can be found in FogROS-G [45].
FogROS2-SGC uses the default DTLS network protocol. We include FogROS2-SGC-TCP
that uses TCP instead of DTLS as a variant.

ROS2 Message Latency: We measure the Round Trip Time (RTT) between when a
robot publishes a ROS2 message to the cloud and when data is received by the
cloud, which echoes a short message on a separate ROS2 topic. The RTT also
includes the time of parsing the messages and analyzing the latency. The
result can be found in Fig. 7. FogROS2-SGC with DTLS has similar performance
as VPN, which has 0.076s round trip latency for small messages. FogROS2-SGC is
10.2% faster than VPN for 8000 byte messages (0.088 vs 0.097). FogROS2-SGC is
19$\times$ faster than rosduct-rosbridge (0.088 vs 1.67). There are two
reasons for this: (1) Rosduct is implemented in Python and provably slower
than Rust. It uses blocking network operations while FogROS2-SGC uses non-
blocking asynchronous network operations for sending and receiving data. (2)
Rosbridge requires seralization of binary messages in JSON, which require more
bytes and lead to larger messages.

Figure 7: Message round trip latency to the cloud (lower is better). Latency
is averaged over more than 50 packet window. FogROS2-SGC is 19 times faster
than rosbridge baseline for 8000 byte message.

ROS2 Message Throughput: Message throughput is measured by the number of
messages processed per second. Different from other experiments, throughput is
measured on the local area network connected with Ethernet, in order to
prevent network bandwidth from being the bottleneck. Table I shows the message
processing throughput. FogROS2-SGC achieves near-native throughput as ROS2 and
incurs only 3% overhead due to the security and conversion to a unified
message format. FogROS2-SGC has 2.1$\times$ higher throughput than rosbridge,
because rosbridge requires more bytes to serialize binary strings.

Protocol | Throughput (msg/second)
---|---
Original ROS | 330.43
SROS2 | 320.17
Rosduct-Rosbridge | 152.79
FogROS2-SGC-TCP | 268.03
FogROS2-SGC | 320.40
TABLE I: Message throughput evaluation of FogROS2-SGC (higher is better).
Every message is 1000 bytes. The throughput of FogROS2-SGC is near native
performance while adding secure and global connectivity and 2.1 times higher
than rosbridge.

Startup and Advertisement Time: In a RIB that has 10,000 routing records, the
average time for publishing a name to the RIB takes 4ms and subscribing to a
name from RIB takes 2ms. The average startup time from starting a program to
receiving the first message takes 2.4 ms.

###  V-B Cloud Robotics Application Benchmarks

We evaluate the network latency of FogROS2-SGC with 3 example cloud robotics
applications: SLAM with ORB-SLAM2 [46], Grasp Planning with Dex-Net [47], and
Motion Planning with Motion Planning Templates (MPT) [48]. The detailed
description of these benchmarks can be found in [18].

As detailed in Table II, although FogROS2-SGC can scale to multiple robots and
provide fine grained access control for the robots, it demonstrates even
better point-to-point performance than VPN in the vSLAM and grasp planning
experiments. FogROS2-SGC is 9.42 times faster than rosbridge-rosduct on
compressed grasp planning images. However, FogROS2-SGC cannot reliably
transmit large and uncompressed grasp planning matrices. The raw matrix after
serialization is larger than 13MB. We observe a significant amount of lost and
out of order messages because the default transport protocol of FogROS2-SGC is
DTLS over UDP and the communication channel does not recover from lost and out
of order messages. Although transmitting such large message within single ROS2
message is rare, users can choose other supported transport protocols (such as
TCP, gRPC) to meet the requirement of their applications.

| vSLAM | Grasp Planning | Motion Planning
---|---|---|---
Scenario |  fr1/xyz1 |  fr1/loop |  raw matrix |  Compressed |  Apartment |  Cubicle
rosduct-rosbridge | 10.31 | 10.29 | 20.3 | 13.67 | 0.08 | 0.08
VPN | 1.16 | 1.45 | 5.7 | 1.47 | 0.07 | 0.07
FogROS2-SGC-TCP | 1.19 | 1.57 | 8.4 | 1.58 | 0.07 | 0.07
FogROS2-SGC | 1.15 | 1.42 | - | 1.45 | 0.07 | 0.07

TABLE II: Network latency of FogROS2-SGC on cloud robotics applications (lower
is better) FogROS2-SGC is better than rosduct-rosbridge and VPN on vSLAM and
compressed grasp planning. We conducted motion planning on other scenarios
(Home, TwistyCool) and the latency is the same.

###  V-C Case Study: Fleet-DAgger

Figure 8: The experiment setup of Fleet-DAgger. Two ABB YuMi robots located in
two separate buildings in Berkeley utilize computation from a server located
in Pittsburgh for an image based block pushing task.

We apply FogROS2-SGC to the control of a fleet of 4 physical robot arms, an
increasingly relevant setting in robotics and the third motivating example in
Fig. 1. We use the physical experiment setup from Fleet-DAgger [4], where each
robot simultaneously performs an image-based block-pushing task (see Fig. 1C).
The task is to repeatedly push a cube to a goal region randomly generated in
the image, where a new goal is sampled from the reachable workspace upon
reaching the previous goal. The 4 workspaces have an identical setup (but
different block positions and goals) to enable the aggregation of each robot’s
data into a shared dataset and training of a single shared policy on this
dataset, as is typical in fleet learning [4]. When autonomous control is
unreliable, the robots fall back on and learn from remote human teleoperation,
where global connectivity can dramatically increase the number of available
humans. The arms belong to two bimanual ABB YuMi robots in two different labs
about $1\text{\,}\mathrm{km}$ apart with separate local area networks. To test
global connectivity, compute is off-loaded to a separate node in a third local
area network at Carnegie Mellon University $3600\text{\,}\mathrm{km}$ away,
where the robot nodes send images of the current state and receive actions to
execute.

In a previous implementation, Hoque et al. [4] use Secure Shell (SSH) and
Secure File Transfer Protocol (SFTP) to communicate between robots and the
centralized compute node and Python multiprocessing to enable simultaneous
execution. This approach requires storing all SSH credentials at a single node
(a security concern), writing image data to the file system of all nodes at
every timestep, complex asynchronous programming, and restricting all node
locations to within the university campus firewall. To mitigate these issues,
we (1) re-implement the communication system with ROS2 and (2) seamlessly
connect all nodes with FogROS2-SGC with TCP by modifying only a single
configuration text file on each node. Relative to the previous implementation,
the FogROS2-SGC implementation reduces communication time by 64% (Table III),
where communication time includes image transmission latency and
synchronization across all arms but not machine learning or arm motion.
FogROS2-SGC also reduces communication time by 33% relative to the initial
implementation even when the robots are in Berkeley, CA and the server is
moved to Pittsburgh, PA. Note that the SSH method does not work between
Berkeley and Pittsburgh due to university network firewalls [49]. A diagram of
the system architecture is in Fig. 8.

Communication System | Server Location | Communication Time (s)
---|---|---
SSH + SFTP | Berkeley, CA | 0.86
Pittsburgh, PA | -
FogROS2-SGC | Berkeley, CA | 0.31
Pittsburgh, PA | 0.58
TABLE III: Communication time of SSH+SFTP and FogROS2-SGC (lower is better).
FogROS2-SGC with TCP reduces the communication time per experiment step (i.e.,
one simultaneous action on the 4 arms) by 64% when compared to SSH+SFTP, and
has 33% lower communication time than SSH+SFTP in Berkeley even if the server
is moved to Pittsburgh. SSH does not work if the server is in Pittsburgh due
to a university firewall restriction.

##  VI Conclusions and Future Work

We present FogROS2-SGC, an extension of FogROS2 that securely connects
robotics components across different physical locations and networks. One
limitation of FogROS2-SGC is that users are unable to use retransmission and
Quality-of-Service (QoS) mechanisms provided by DDS for inter-ROS2 network
communication. However, users can flexibly choose any supported transport
protocol (e.g., TCP and gRPC). FogROS2-SGC will support other ROS2
communication patterns (e.g., services and actions) in the future.

## Acknowledgement

This research was performed at the AUTOLAB at UC Berkeley in affiliation with
the Berkeley AI Research (BAIR) Lab. The authors were supported in part by
donations from Bosch, Toyota Research Institute, Google, Siemens, and Autodesk
and by equipment grants from PhotoNeo, NVidia, and Intuitive Surgical. The
research is also supported by C3.ai Digital Transformation Institute for AI
resilience.

## References

  * [1] B. Kehoe, S. Patil, P. Abbeel, and K. Goldberg, “A survey of research on cloud robotics and automation,” _IEEE Trans. Automation Science and Engineering_ , vol. 12, no. 2, pp. 398–409, 2015.
  * [2] J. Ichnowski, K. Chen, K. Dharmarajan, S. Adebola, M. Danielczuk, V. Mayoral-Vilches, H. Zhan, D. Xu, R. Ghassemi, J. Kubiatowicz, _et al._ , “FogROS2: An adaptive and extensible platform for cloud and fog robotics using ros 2,” _arXiv preprint arXiv:2205.09778_ , 2022.
  * [3] “FogROS2 Official ROS2 Repository,” https://index.ros.org/p/fogros2/.
  * [4] R. Hoque, L. Y. Chen, S. Sharma, K. Dharmarajan, B. Thananjeyan, P. Abbeel, and K. Goldberg, “Fleet-DAgger: Interactive robot fleet learning with scalable human supervision,” _Conference on Robot Learning (CoRL)_ , 2022.
  * [5] S. Sumikura, M. Shibuya, and K. Sakurada, “OpenVSLAM: A versatile visual slam framework,” in _Proceedings of the 27th ACM International Conference on Multimedia_ , ser. MM ’19. New York, NY, USA: ACM, 2019, pp. 2292–2295. [Online]. Available: http://doi.acm.org/10.1145/3343031.3350539
  * [6] A. S. Eddington, _The mathematical theory of relativity_. The University Press, 1923.
  * [7] V. Mayoral-Vilches, R. White, G. Caiazza, and M. Arguedas, “SROS2: Usable cyber security tools for ros 2,” _arXiv e-prints_ , pp. arXiv–2208, 2022\.
  * [8] “rosduct,” https://github.com/uts-magic-lab/rosduct.
  * [9] C. Crick, G. Jay, S. Osentoski, and O. C. Jenkins, “ROS and Rosbridge: Roboticists out of the loop,” in _2012 7th ACM/IEEE International Conference on Human-Robot Interaction (HRI)_ , 2012, pp. 493–494.
  * [10] “Scalable Distributed Robot Fleet With Fast DDS Discovery Server,” https://husarnet.com/blog/ros2-dds-discovery-server, accessed: 2023-03-1.
  * [11] N. Tian, M. Matl, J. Mahler, Y. X. Zhou, S. Staszak, C. Correa, S. Zheng, Q. Li, R. Zhang, and K. Goldberg, “A cloud robot system using the dexterity network and Berkeley robotics and automation as a service (BRASS),” in _Proc. IEEE Int. Conf. Robotics and Automation (ICRA)_ , 2017, pp. 1615–1622.
  * [12] B. Kehoe, A. Matsukawa, S. Candido, J. Kuffner, and K. Goldberg, “Cloud-based robot grasping with the google object recognition engine,” in _Proc. IEEE Int. Conf. Robotics and Automation (ICRA)_ , 2013, pp. 4263–4270.
  * [13] P. Li, B. DeRose, J. Mahler, J. A. Ojea, A. K. Tanwani, and K. Goldberg, “Dex-Net as a service (DNaaS): A cloud-based robust robot grasp planning system,” in _Proc. IEEE Conf. on Automation Science and Engineering (CASE)_ , 2018, pp. 1420–1427.
  * [14] B. Kehoe, D. Berenson, and K. Goldberg, “Estimating part tolerance bounds based on adaptive cloud-based grasp planning with slip,” in _Proc. IEEE Conf. on Automation Science and Engineering (CASE)_ , 2012, pp. 1106–1113.
  * [15] B. Kehoe, D. Berenson, and G. Ken, “Toward cloud-based grasping with uncertainty in shape: Estimating lower bounds on achieving force closure with zero-slip push grasps,” in _Proc. IEEE Int. Conf. Robotics and Automation (ICRA)_ , 2012, pp. 576–583.
  * [16] B. Kehoe, D. Warrier, S. Patil, and K. Goldberg, “Cloud-based grasp analysis and planning for toleranced parts using parallelized Monte Carlo sampling,” _IEEE Trans. Automation Science and Engineering_ , vol. 12, no. 2, pp. 455–470, 2014.
  * [17] M.-L. Lam and K.-Y. Lam, “Path planning as a service PPaaS: Cloud-based robotic path planning,” in _Proc. IEEE Int. Conf. on Robotics and Biomimetics (ROBIO)_ , 2014, pp. 1839–1844.
  * [18] K. E. Chen, Y. Liang, N. Jha, J. Ichnowski, M. Danielczuk, J. Gonzalez, J. Kubiatowicz, and K. Goldberg, “FogROS: An adaptive framework for automating fog robotics deployment,” in _2021 IEEE 17th International Conference on Automation Science and Engineering (CASE)_. IEEE, 2021, pp. 2035–2042.
  * [19] J. Ichnowski, W. Lee, V. Murta, S. Paradis, R. Alterovitz, J. E. Gonzalez, I. Stoica, and K. Goldberg, “Fog robotics algorithms for distributed motion planning using lambda serverless computing,” in _Proc. IEEE Int. Conf. Robotics and Automation (ICRA)_ , 2020, pp. 4232–4238.
  * [20] R. Anand, J. Ichnowski, C. Wu, J. M. Hellerstein, J. E. Gonzalez, and K. Goldberg, “Serverless multi-query motion planning for fog robotics,” in _Proc. IEEE Int. Conf. Robotics and Automation (ICRA)_. IEEE, 2021.
  * [21] G. McGrath and P. R. Brenner, “Serverless computing: Design, implementation, and performance,” in _2017 IEEE 37th International Conference on Distributed Computing Systems Workshops (ICDCSW)_. IEEE, 2017, pp. 405–410.
  * [22] G. Swamy, S. Reddy, S. Levine, and A. D. Dragan, “Scaled autonomy: Enabling human operators to control robot fleets,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2020, pp. 5942–5948.
  * [23] A. K. Tanwani, R. Anand, J. E. Gonzalez, and K. Goldberg, “RILaaS: Robot inference and learning as a service,” _IEEE Robotics & Automation Letters_, vol. 5, no. 3, pp. 4423–4430, 2020.
  * [24] S. Bauer, F. Widmaier, M. Wüthrich, N. Funk, J. U. D. Jesus, J. Peters, J. Watson, C. Chen, K. Srinivasan, J. Zhang, J. Zhang, M. R. Walter, R. Madan, C. B. Schaff, T. Maeda, T. Yoneda, D. Yarats, A. Allshire, E. K. Gordon, T. Bhattacharjee, S. S. Srinivasa, A. Garg, A. Buchholz, S. Stark, T. Steinbrenner, J. Akpo, S. Joshi, V. Agrawal, and B. Schölkopf, “A robot cluster for reproducible research in dexterous manipulation,” _arXiv preprint arXiv:2109.10957_ , 2021.
  * [25] J. Z. Lim and D. W.-K. Ng, “Cloud based implementation of ROS through VPN,” in _Int. Conf. on Smart Computing & Communications (ICSCC)_. IEEE, 2019, pp. 1–5.
  * [26] S. S. H. Hajjaj and K. S. M. Sahari, “Establishing remote networks for ROS applications via port forwarding: A detailed tutorial,” _International Journal of Advanced Robotic Systems_ , vol. 14, no. 3, p. 1729881417703355, 2017\.
  * [27] eProsima, “Fast DDS,” https://www.eprosima.com/index.php/products-all/eprosima-fast-dds.
  * [28] “RTI Connext DDS,” https://www.rti.com/products.
  * [29] “Integrating ROS2 with Eclipse zenoh,” https://zenoh.io/blog/2021-04-28-ros2-integration/, accessed: 2021-02-15.
  * [30] A. B. M. Pereira, R. E. Julio, and G. S. Bastos, “ROSRemote: Using ROS on cloud to access robots remotely,” in _Robot Operating System (ROS)_. Springer, 2019, pp. 569–605.
  * [31] B. Xu and J. Bian, “A cloud robotic application platform design based on the microservices architecture,” in _Int. Conf. on Control, Robotics and Intelligent System_ , 2020, pp. 13–18.
  * [32] E. Boasson, A. Corsaro, and H. van t Hag, “Eclipse Cyclone DDS,” https://projects.eclipse.org/projects/iot.cyclonedds.
  * [33] “Google Remote Procedure Call,” https://grpc.io/.
  * [34] M. Myers, C. Adams, D. Solo, and D. Kemp, “Internet x. 509 certificate request message format,” Tech. Rep., 1999.
  * [35] “Signature and hash algorithms for TLS and DTLS,” https://www.ibm.com/docs/en/zos/2.5.0?topic=support-signature-hash-algorithms.
  * [36] A. W. Appel, “Verification of a cryptographic primitive: Sha-256,” _ACM Transactions on Programming Languages and Systems (TOPLAS)_ , vol. 37, no. 2, pp. 1–31, 2015.
  * [37] N. Mor, R. Pratt, E. Allman, K. Lutz, and J. Kubiatowicz, “Global data plane: A federated vision for secure data in edge computing,” in _2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS)_. IEEE, 2019, pp. 1652–1663.
  * [38] “Multicast DNS RFC 6762,” https://www.rfc-editor.org/rfc/rfc6762.html.
  * [39] J. Daemen and V. Rijmen, “AES proposal: Rijndael,” 1999.
  * [40] “RSASSA-PSS RFC 4056,” https://www.rfc-editor.org/rfc/rfc4056.
  * [41] N. D. Matsakis and F. S. Klock II, “The rust language,” in _ACM SIGAda Ada Letters_ , vol. 34, no. 3. ACM, 2014, pp. 103–104.
  * [42] “Tokio,” https://tokio.rs/.
  * [43] “wireguard VPN,” https://www.wireguard.com/.
  * [44] A. Panda, S. Han, K. Jang, M. Walls, S. Ratnasamy, and S. Shenker, “$\\{$NetBricks$\\}$: Taking the v out of $\\{$NFV$\\}$,” in _12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16)_ , 2016, pp. 203–216.
  * [45] K. Chen, J. Yuan, N. Jha, J. Ichnowski, J. Kubiatowicz, and K. Goldberg, “FogROS G: Enabling secure, connected and mobile fog robotics with global addressability,” _arXiv preprint arXiv:2210.11691_ , 2022.
  * [46] R. Mur-Artal and J. D. Tardós, “ORB-SLAM2: An open-source slam system for monocular, stereo, and RGB-D cameras,” _IEEE Trans. Robotics_ , vol. 33, no. 5, pp. 1255–1262, 2017.
  * [47] J. Mahler, J. Liang, S. Niyaz, M. Laskey, R. Doan, X. Liu, J. A. Ojea, and K. Goldberg, “Dex-Net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics,” in _Proc. Robotics: Science and Systems (RSS)_ , 2017.
  * [48] J. Ichnowski and R. Alterovitz, “Motion planning templates: A motion planning framework for robots with low-power CPUs,” in _Proc. IEEE Int. Conf. Robotics and Automation (ICRA)_ , 2019.
  * [49] “UC Berkeley Minimum Security Standard,” https://security.berkeley.edu/policy/minimum-security-standards-networked-devices-mssnd.


1Harvard University, 2Adobe Research
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
# Smoothed Online Combinatorial Optimization Using Dynamic Planning Windows
and Its Application in Apache Kafka

Kai Wang,1 Zhao Song, 2 Georgios Theocharous 2, Sridhar Mahadevan2

###### Abstract

Smoothed online combinatorial optimization is an extension of online convex
optimization where the objective function can be combinatorial and the learner
incurs a penalty for changing the decisions in consecutive rounds. Smoothed
online combinatorial optimization can be applied to distributed streaming
system, e.g., Apache Kafka, where a bipartite matching between streaming
topics and servers needs to be maintained to process the varying streaming
data with the goal of minimizing a server imbalance cost and a switching cost
associated to every matching assignment change. This paper studies smoothed
online combinatorial optimization problem with a predictive model, where the
model can predict the future cost functions with uncertainty. We show that
using predictions to plan can produce a regret dependent on the total
uncertainty and an additional regret for every planning restart. Since the
predictive uncertainty can vary, we propose to use a dynamic planning window
to balance the total predictive uncertainty and the number of planning
windows, which results in an upper bound and a lower bound on the regret. To
resolve the combinatorial challenge, we provide an iterative algorithm to
decouple the temporal dependency caused by switching cost. Empirically, our
algorithm outperforms existing baselines with a smaller regret in the
distributed streaming problem using synthetic streaming traffic.