Lipschitzian Estimates in Discrete-Time Constrained Stochastic Optimal Control

In the last decades, control problems with infinite horizons and discount factors have become increasingly central not only for economics but also for applications in artificial intelligence and machine learning. The strong links between reinforcement learning and control theory have led to major efforts towards the development of algorithms to learn how to solve constrained control problems. In particular, discount plays a role in addressing the challenges that come with mod...

We investigate constrained optimal control problems for linear stochastic dynamical systems evolving in discrete time. We consider minimization of an expected value cost over a finite horizon. Hard constraints are introduced first, and then reformulated in terms of probabilistic constraints. It is shown that, for a suitable parametrization of the control policy, a wide class of the resulting optimization problems are convex, or admit reasonable convex approximations.

We study the optimal value function for control problems on Banach spaces that involve both continuous and discrete control decisions. For problems involving semilinear dynamics subject to mixed control inequality constraints, one can show that the optimal value depends locally Lipschitz continuously on perturbations of the initial data and the costs under rather natural assumptions. We prove a similar result for perturbations of the initial data, the constraints and the cost...

In this paper we present a dynamic programing approach to stochastic optimal control problems with dynamic, time-consistent risk constraints. Constrained stochastic optimal control problems, which naturally arise when one has to consider multiple objectives, have been extensively investigated in the past 20 years, however, in most formulations, the constraints are formulated as either risk-neutral (i.e., by considering an expected cost), or by applying static, single-period r...

We obtain a probabilistic proof of the local Lipschitz continuity for the optimal stopping boundary of a class of problems with state space $[0,T]\times\mathbb{R}^d$, $d\ge 1$. To the best of our knowledge this is the only existing proof that relies exclusively upon stochastic calculus, all the other proofs making use of PDE techniques and integral equations. Thanks to our approach we obtain our result for a class of diffusions whose associated second order differential opera...

The main goal of this paper is to apply the machinery of variational analysis and generalized differentiation to study infinite horizon stochastic dynamic programming (DP) with discrete time in the Banach space setting without convexity assumptions. Unlike to standard stochastic DP with stationary Markov processes, we investigate here stochastic DP in $L^p$ spaces to deal with nonstationary stochastic processes, which describe a more flexible learning procedure for the decisi...

We study optimal control problems governed by abstract infinite dimensional stochastic differential equations using the dynamic programming approach. In the first part, we prove Lipschitz continuity, semiconcavity and semiconvexity of the value function under several sets of assumptions, and thus derive its $C^{1,1}$ regularity in the space variable. Based on this regularity result, we construct optimal feedback controls using the notion of the $B$-continuous viscosity soluti...

In this paper we present explicit estimate for Lipschitz constant of solution to a problem of calculus of variations. The approach we use is due to Gamkrelidze and is based on the equivalence of the problem of calculus of variations and a time-optimal control problem. The obtained estimate is used to compute complexity bounds for a path-following method applied to a convex problem of calculus of variations with polyhedral end-point constraints.

In this paper, we consider the finite-state approximation of a discrete-time constrained Markov decision process (MDP) under the discounted and average cost criteria. Using the linear programming formulation of the constrained discounted cost problem, we prove the asymptotic convergence of the optimal value of the finite-state model to the optimal value of the original model. With further continuity condition on the transition probability, we also establish a method to comput...

In this paper, we present a discretization algorithm for finite horizon risk constrained dynamic programming algorithm in [Chow_Pavone_13]. Although in a theoretical standpoint, Bellman's recursion provides a systematic way to find optimal value functions and generate optimal history dependent policies, there is a serious computational issue. Even if the state space and action space of this constrained stochastic optimal control problem are finite, the spaces of risk threshol...