Bounded-Velocity Stochastic Control for Dynamic Resource Allocation

The stochastic knapsack has been used as a model in wide ranging applications from dynamic resource allocation to admission control in telecommunication. In recent years, a variation of the model has become a basic tool in studying problems that arise in revenue management and dynamic/flexible pricing; and it is in this context that our study is undertaken. Based on a dynamic programming formulation and associated properties of the value function, we study in this paper a cla...

The paper introduces the first formulation of convex Q-learning for Markov decision processes with function approximation. The algorithms and theory rest on a relaxation of a dual of Manne's celebrated linear programming characterization of optimal control. The main contributions firstly concern properties of the relaxation, described as a deterministic convex program: we identify conditions for a bounded solution, and a significant relationship between the solution to the ne...

Consider a storage system where the content is driven by a Brownian motion absent control. At any time, one may increase or decrease the content at a cost proportional to the amount of adjustment. A decrease of the content takes effect immediately, while an increase is realized after a fixed lead time $\lt$. Holding costs are incurred continuously over time and are a convex function of the content. The objective is to find a control policy that minimizes the expected present ...

We deal with an infinite horizon, infinite dimensional stochastic optimal control problem arising in the study of economic growth in time-space. Such problem has been the object of various papers in deterministic cases when the possible presence of stochastic disturbances is ignored. Here we propose and solve a stochastic generalization of such models where the stochastic term, in line with the standard stochastic economic growth models, is a multiplicative one, driven by a c...

This paper proposes a new formulation for the dynamic resource allocation problem, which converts the traditional MDP model with known parameters and no capacity constraints to a new model with uncertain parameters and a resource capacity constraint. Our motivating example comes from a medical resource allocation problem: patients with multiple chronic diseases can be provided either normal or special care, where the capacity of special care is limited due to financial or hum...

In this paper we present a dynamic programing approach to stochastic optimal control problems with dynamic, time-consistent risk constraints. Constrained stochastic optimal control problems, which naturally arise when one has to consider multiple objectives, have been extensively investigated in the past 20 years, however, in most formulations, the constraints are formulated as either risk-neutral (i.e., by considering an expected cost), or by applying static, single-period r...

We consider an online resource allocation problem where multiple resources, each with an individual initial capacity, are available to serve random requests arriving sequentially over multiple discrete time periods. At each time period, one request arrives and its associated reward and size are drawn independently from a known distribution that could be resource-dependent. Upon its arrival and revealing itself, an online decision has to be made on whether or not to accept the...

We consider the classical problem of sequential resource allocation where a decision maker must repeatedly divide a budget between several resources, each with diminishing returns. This can be recast as a specific stochastic optimization problem where the objective is to maximize the cumulative reward, or equivalently to minimize the regret. We construct an algorithm that is {\em adaptive} to the complexity of the problem, expressed in term of the regularity of the returns of...

These lecture notes are derived from a graduate-level course in dynamic optimization, offering an introduction to techniques and models extensively used in management science, economics, operations research, engineering, and computer science. The course emphasizes the theoretical underpinnings of discrete-time dynamic programming models and advanced algorithmic strategies for solving these models. Unlike typical treatments, it provides a proof for the principle of optimality ...

We apply numerical dynamic programming techniques to solve discrete-time multi-asset dynamic portfolio optimization problems with proportional transaction costs and shorting/borrowing constraints. Examples include problems with multiple assets, and many trading periods in a finite horizon problem. We also solve dynamic stochastic problems, with a portfolio including one risk-free asset, an option, and its underlying risky asset, under the existence of transaction costs and co...