Risk-sensitive robot navigation

[Risk-conditioned distributional soft actor-critic for risk-sensitive navigation. J. Choi, C. Dance et al. ICRA 2021.]

We aim to design robot navigation policies that optimize risk-averse objectives. We say an objective is risk averse if it puts more weight on bad outcomes than on good ones. Optimizing these objectives is interesting for several reasons. First, risk-averse policies may be safer than risk-neutral policies, while avoiding the over-cautious behaviour typical of policies based on worst-case analyses (1). Second, in environments that are so complex they’re impractical to model accurately, risk-averse policies provide guarantees on robustness to modelling errors (2). Third, navigation agents are designed, insured and used by us risk-averse humans (3), making risk-averse objectives a natural choice.

A navigation policy should quickly get a robot to a given place, reacting in real-time to moving people. It should not harm anything, get lost, or require an accurate map. If the policy is the output of a reinforcement learning (RL) algorithm, that algorithm should optimize a risk-averse objective, not just the sum of rewards like most RL algorithms. Preferably, this risk-averse objective should be a coherent risk measure (1) as these measures care not only about “how often bad things happen” but also about “how bad the bad outcomes are”. Many traditional objectives like value-at-risk and chance-constrained objectives are not coherent. Finally, as it can often take hours to train a policy with RL, it should be possible to tune the policy’s behaviour for different environments, without needing to retrain it. In particular, we’d like to be able to tune the degree of risk-aversion of a single RL policy, from very cautious behaviours in the presence of small children, to ultra-efficient behaviours for robot-only zones of warehouses.

Related work

The literature on classical model-predictive-control (MPC) approaches to robot navigation has explored a wide variety of risk-averse objectives, ranging from chance constraints (4) to entropic risk (5). However, the only such work that uses a coherent risk measure is (6). Recently, deep RL approaches to robot navigation have become popular. Unlike MPC methods, RL can infer optimal actions without computationally expensive trajectory predictions, and can directly address complex reward functions and simulated motion models (7, 8). Several such deep-RL navigation methods explicitly account for risks due to uncertainties about the environment, resulting in improved safety (9, 10, 11, 12). However, such methods require additional prediction models, carefully shaped reward functions, or expensive Monte Carlo sampling at run-time. Moreover, they don’t optimize coherent risk measures, which is not even possible with conventional RL algorithms. While so-called distributional RL algorithms, which can optimize such risk measures, were proposed over a decade ago (13), these algorithms have only become popular since they were coupled with deep learning (14). Previously, distributional RL has only been applied to robotics in simulation (15) or for real-world grasping tasks (16).

Our approach to adaptable risk aversion

Our overall idea is to train a risk-conditioned policy, which is a policy that takes as input a description of how cautiously it should behave. We train these policies in a simulation environment that models a wheeled robot with a range sensor, which has to navigate around walls and other objects. At each time, the policy must choose the robot’s desired linear and angular velocities, and the environment supplies the agent with a reward. The reward encourages the agent to follow a given sequence of waypoints efficiently, while avoiding collisions and preferring to move in straight lines rather than waggling from side to side. The sum of rewards over time is called the return but, whilst conventional RL simply aims to maximize the average return with respect to the policy, our work aims to maximize a risk-averse objective, defined as follows.

Risk-Averse Objective. Let  be the return of a policy  from some given initial state. The return  is a real-valued random variable. For a quantile fraction , consider its quantile function given by . Our risk-averse objective is then defined as  where  is a mapping from quantile fractions to quantile fractions, called a distortion function.

We consider two specific families of risk-averse objectives, given by different choices of distortion function, each having a scalar parameter . The first is the conditional value-at-risk (CVaR), which is the expectation of the fraction  of least-favourable returns and corresponds to the distortion function , for . Figure 1 illustrates how the use of the CVaR risk-averse objective influences decision-making. The second family is the power-law risk measure, which corresponds to the distortion function , for . Figure 1 illustrates how the use of the CVaR risk-averse objective influences decision-making. The second family is the power-law risk measure, which corresponds to the distortion function , for  > 0. For both objectives, lower values of  corresponds to more cautious (i.e. risk-averse) behaviour. Both objectives are coherent risk measures, and we chose them for their simplicity and because they have been used by other authors.

RL Algorithm. Our algorithm is based on the distributional soft actor-critic (DSAC) algorithm of Ma et al. (17) which couples the soft actor-critic (SAC) algorithm (20) with the implicit quantile network (IQN) algorithm (14). SAC substantially speeds up learning by working with a soft return, given by adding a multiple of the entropy of the action distribution to the reward in each state, and by using a corresponding stochastic policy. While SAC maximizes the average of this soft return, DSAC plugs the soft return into IQN to learn the quantile function of the soft return. As explained above, our risk-averse objective is a simple expectation of this quantile function for an appropriate distortion function . We can optimize this expectation with respect to the policy using (KL-minimization) ideas from SAC.

The main novel feature of our RL algorithm, which we call risk-conditioned distributional soft actor-critic (RC-DSAC), is to allow the user to switch between policies with different degrees of risk-aversion at deployment time, without the need for retraining. To achieve this, we provide the parameter  of the risk distortion function as an input to both the policy and the estimated quantile function. Directly inputting as a scalar does not provide a sufficiently rich representation to ensure good generalization over different values. Therefore, we input the cosine embedding of , in the same way that IQN inputs the quantile fraction . Figure 2 shows the resulting network architecture.

Results. We conducted experiments to compare the performance of different RL-based navigation policies, in both simulated and real-world office environments. In this blog, we discuss results for four agents: a SAC agent, which is risk-neutral and maximizes the average reward; an RC-DSAC agent using the distortion function , which takes the risk-aversion parameter as input; and two DSAC agents, each trained with a different value of , also for the distortion function . The full paper (19) presents results for other RL algorithms, sensor configurations and distortion functions. All agents use the same network architecture, except that SAC has no cosine embedding for or , and DSAC has no cosine embedding for . We trained each agent for 5000 episodes in 500 randomly generated environments, each episode lasting for a maximum of 1000 steps.

In the simulation experiment, we ran the algorithms for 10 episodes on each of 50 different environments. These test environments were distinct from the training environments, but we used the same test environments and start/goal states for each algorithm to ensure a fair comparison. Figure 3 shows the results. Our main observations are as follows.

• The mean number of collisions per episode is over 0.15 for all agents. This high collision rate is disappointing, and we attribute it partly to the use of a (simulation of a) sparse arrangement of low-cost sensors.
• DSAC trained with  = 0.25 has a much lower return than other methods. We attribute this to over-cautiousness, and in our real-world experiment, it hesitantly wavers from side to side whenever it approaches an obstacle.
• Comparing SAC with the distributional algorithms, we see that SAC has both a higher collision rate and a comparable or lower return – it is Pareto inferior to the distributional methods (except for DSAC with  = 0.25, as just discussed). It’s previously been observed that distributional RL often outperforms classical average-reward RL, even in risk-neutral settings (14), conjecturing that this might be because predicting quantiles serves as an auxiliary task that enhances representation learning.
• RC-DSAC with = 0.25 has fewer collisions and a lower return than RC-DSAC for higher values of . This is evidence that we’re achieving our original objective of being able to tune the degree of risk-aversion of a single RL policy just by adjusting an input parameter, without any retraining being required.

We also deployed the SAC, DSAC and RC-DSAC agents on a real robot, given the task of navigating 107.4 m through a cluttered office environment. Figure 4 shows a video of the robot’s behaviour. In line with the findings of our simulation experiment, DSAC trained with  = 0.25 is over-cautious and takes the longest to complete the course, while SAC has more collisions than other methods (5 collisions in total versus 0 or 1 for all other agents).

Discussion and future work

There are many avenues for improvement on these results. First, we should like to understand why there were so many collisions. Maybe a change of sensors would be necessary to achieve this. Second, we gave a fixed reward of -10 if there’s a collision, noting that it’s common in the RL literature to perform “reward clipping” to improve algorithm convergence (18). However, it might be more natural to have collision penalties that are more negative for more severe collisions. Furthermore, in addition to evaluating agents in terms of collision rate and average return as we do in the paper, it would be natural to compare them in terms of the risk-averse objectives that they attempt to optimize.

We aim to ensure the safety of RL-based policies for robots operating in environments that are too complex to model accurately, and in applications where strict worst-case analyses result in unacceptably over-cautious behaviours. Distributional RL with coherent risk measures seems a natural way to achieve this. However, we’re only at the beginning of this endeavour and there’s plenty of room for progress on fundamental questions. Which of many different coherent risk measures should we prefer in robotics? How should we address the “time inconsistency” (1) of risk-averse decision-making? How can we improve the extent to which distributional RL optimizes risk-averse objectives? It will also be interesting to apply risk-averse objectives in other situations, for instance to tasks defined by a few human demonstrations, or where multiple robots interact. Our work should be among the first of many to deploy policies trained to optimize risk measures in the real world.

1: How should a robot assess risk? Towards an axiomatic theory of risk in robotics. A. Majumdar and M. Pavone. Robotics Research, Springer, 2020, pp. 75–84.

2: Risk-sensitive and robust decision-making: A CVaR optimization approach. Y. Chow, A. Tamar, S. Mannor, and M. Pavone.  Advances in Neural Information Processing Systems 28 (NIPS), 2015, pp. 1522–1530.

3: Prospect theory: An analysis of decision under risk. D. Kahneman and A. Tversky. Handbook of the Fundamentals of Financial Decision Making: Part I, World Scientific, 2013, pp. 99–127.

4: Chance-constrained optimal path planning with obstacles. L. Blackmore, M. Ono, and B. C. Williams. IEEE Transactions on Robotics, vol. 27, no. 6, pp. 1080–1094, 2011.

5: Risk-sensitive sequential action control with multi-modal human trajectory forecasting for safe crowd-robot interaction. H. Nishimura, B. Ivanovic, A. Gaidon, M. Pavone, and M. Schwager. arXiv preprint arXiv:2009.05702, 2020.

6: Wasserstein distributionally robust motion planning and control with safety constraints using conditional value-at-risk. A. Hakobyan and I. Yang. IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2020, pp. 490–496.

7: Learning navigation behaviors end-to-end with AutoRL. H.-T. L. Chiang, A. Faust, M. Fiser, and A. Francis. IEEE Robotics and Automation Letters, vol. 4, no. 2, pp. 2007–2014, 2019.

8: Fast adaptation of deep reinforcement learning-based navigation skills to human preference. J. Choi, C. Dance, J. Kim, K. Park, J. Han, J. Seo, and M. Kim. IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2020, pp. 3363–3370.

9: Learning resilient behaviors for navigation under uncertainty. T. Fan, P. Long, W. Liu, J. Pan, R. Yang, and D. Manocha. IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2020, pp. 5299–5305.

10: Safe reinforcement learning with model uncertainty estimates. B. Lötjens, M. Everett, and J. P. How. IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2019, pp. 8662–8668.

11: Risk-aware high-level decisions for automated driving at occluded intersections with reinforcement learning. D. Kamran, C. F. Lopez, M. Lauer, and C. Stiller. arXiv preprint arXiv:2004.04450, 2020.

12: Intent-aware pedestrian prediction for adaptive crowd navigation. K. D. Katyal, G. D. Hager, and C.-M. Huang. IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2020, pp. 3277–3283.

13: Nonparametric return distribution approximation for reinforcement learning. T. Morimura, M. Sugiyama, H. Kashima, H. Hachiya, and T. Tanaka. Proceedings of the 27th International Conference on Machine Learning (ICML), 2010, pp. 799–806.

14: Implicit quantile networks for distributional reinforcement learning. W. Dabney, G. Ostrovski, D. Silver, and R. Munos. arXiv preprint arXiv:1806.06923, 2018.

15: Improving robustness via risk averse distributional reinforcement learning. R. Singh, Q. Zhang, and Y. Chenin. Proceedings of the 2nd Conference on Learning for Dynamics and Control, A. M. Bayen, A. Jadbabaie, G. Pappas, P. A. Parrilo, B. Recht, C. Tomlin, and M. Zeilinger, Eds., PMLR 2020, pp. 958–968.

16: Quantile QT-Opt for risk-aware vision-based robotic grasping. C. Bodnar, A. Li, K. Hausman, P. Pastor, and M. Kalakrishnan, In Robotics: Science and Systems (RSS), 2020.

17: DSAC: Distributional soft actor critic for risk-sensitive reinforcement learning. X. Ma, L. Xia, Z. Zhou, J. Yang, and Q. Zhao. arXiv preprint arXiv:1710.10044, 2020.

18: Is deep reinforcement learning really superhuman on Atari? Leveling the playing field. M. Toromanoff, E. Wirbel, F. Moutarde. arXiv preprint arXiv:1908.04683, 2019.

19: Risk-conditioned distributional soft actor-critic for risk-sensitive navigation. J. Choi, C. Dance, J. Kim, S. Hwang, K. Park. IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2021.

20: Acquiring diverse robot skills via maximum entropy deep reinforcement learning. T. Haarnoja. Technical Report No. UCB/EECS-2018-176, 2018.