Seth A, Hicks JL, Uchida TK, Habib A, Dembia CL, Dunne JJ, Ong CF, DeMers MS, Rajagopal A, Millard M, et al. Opensim: Simulating musculoskeletal dynamics and neuromuscular control to study human and animal movement. PLoS Comput Biol. 2018;14(7).

Dembia CL, Bianco NA, Falisse A, Hicks JL, Delp SL. Opensim moco: Musculoskeletal optimal control. PLoS Comput Biol. 2019;16(12).

Todorov E, Erez T, Mujoco TY. A physics engine for model-based control. In: 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2012;5026–33.

Lee J, Grey M, Ha S, Kunz T, Jain S, Ye Y, Srinivasa S, Stilman M, Liu C. Dart: Dynamic animation and robotics toolkit. J Open Source Softw. 2018;3(22):500.

Article
Google Scholar

Hwangbo J, Lee J, Hutter M. Per-contact iteration method for solving contact dynamics. IEEE Robot Autom Lett. 2018;3(2):895–902.

Article
Google Scholar

Song S, Geyer H. A neural circuitry that emphasizes spinal feedback generates diverse behaviours of human locomotion. J Physiol. 2015;593(16):3493–511.

Article
CAS
PubMed
PubMed Central
Google Scholar

Ong CF, Geijtenbeek T, Hicks JL, Delp SL. Predicting gait adaptations due to ankle plantarflexor muscle weakness and contracture using physics-based musculoskeletal simulations. PLoS Comput Biol. 2019;15(10).

Tamura D, Aoi S, Funato T, Fujiki S, Senda K, Tsuchiya K. Contribution of phase resetting to adaptive rhythm control in human walking based on the phase response curves of a neuromusculoskeletal model. Front Neurosci. 2020;14:17.

Article
PubMed
PubMed Central
Google Scholar

Richards BA, Lillicrap TP, Beaudoin P, Bengio Y, Bogacz R, Christensen A, Clopath C, Costa RP, de Berker A, Ganguli S, et al. A deep learning framework for neuroscience. Nature Neurosci. 2019;22(11):1761–70.

Article
CAS
PubMed
Google Scholar

Peng XB, Abbeel P, Levine S, van de Panne M. Deepmimic: Example-guided deep reinforcement learning of physics-based character skills. ACM Trans Graph. 2018;37(4):1–14.

Article
CAS
Google Scholar

Lee S, Park M, Lee K, Lee J. Scalable muscle-actuated human simulation and control. ACM Trans Graph. 2019;38(4):1–13.

Article
Google Scholar

Anand AS, Zhao G, Roth H, Seyfarth A. A deep reinforcement learning based approach towards generating human walking behavior with a neuromuscular model. In: 2019 IEEE-RAS 19th International Conference on Humanoid Robots (Humanoids), 2019;537–43.

Delp SL, Loan JP, Hoy MG, Zajac FE, Topp EL, Rosen JM. An interactive graphics-based model of the lower extremity to study orthopaedic surgical procedures. IEEE Trans Biomed Eng. 1990;37(8):757–67.

Article
CAS
PubMed
Google Scholar

Seth A, Dong M, Matias R, Delp SL. Muscle contributions to upper-extremity movement and work from a musculoskeletal model of the human shoulder. Front Neurorobot. 2019;13:90.

Article
PubMed
PubMed Central
Google Scholar

Lerner ZF, DeMers MS, Delp SL, Browning RC. How tibiofemoral alignment and contact locations affect predictions of medial and lateral tibiofemoral contact forces. J Biomech. 2015;48(4):644–50.

Article
PubMed
PubMed Central
Google Scholar

Arnold EM, Ward SR, Lieber RL, Delp SL. A model of the lower limb for analysis of human movement. Ann Biomed Eng. 2010;38(2):269–79.

Article
PubMed
Google Scholar

Rajagopal A, Dembia CL, DeMers MS, Delp DD, Hicks JL, Delp SL. Full-body musculoskeletal model for muscle-driven simulation of human gait. IEEE Trans Biomed Eng. 2016;63(10):2068–79.

Article
PubMed
PubMed Central
Google Scholar

Hill AV. The heat of shortening and the dynamic constants of muscle. Proc R Soc London. 1938;126(843):136–95.

Google Scholar

Zajac FE. Muscle and tendon: properties, models, scaling, and application to biomechanics and motor control. Crit Rev Biomed Eng. 1989;17(4):359–411.

CAS
PubMed
Google Scholar

Geyer H, Herr H. A muscle-reflex model that encodes principles of legged mechanics produces human walking dynamics and muscle activities. IEEE Trans Neural Syst Rehab Eng. 2010;18(3):263–73.

Article
Google Scholar

Millard M, Uchida T, Seth A, Delp SL. Flexing computational muscle: modeling and simulation of musculotendon dynamics. J Biomech Eng. 2013;135:2.

Article
Google Scholar

Haeufle D, Günther M, Bayer A, Schmitt S. Hill-type muscle model with serial damping and eccentric force-velocity relation. J Biomech. 2014;47(6):1531–6.

Article
CAS
PubMed
Google Scholar

Bhargava LJ, Pandy MG, Anderson FC. A phenomenological model for estimating metabolic energy consumption in muscle contraction. J Biomech. 2004;37(1):81–8.

Article
PubMed
Google Scholar

Umberger BR. Stance and swing phase costs in human walking. J R Soc Interface. 2010;7(50):1329–40.

Article
PubMed
PubMed Central
Google Scholar

Koelewijn AD, Heinrich D, Van Den Bogert AJ. Metabolic cost calculations of gait using musculoskeletal energy models, a comparison study. PLoS ONE. 2019;14:9.

Article
CAS
Google Scholar

Ackermann M, Van den Bogert AJ. Optimality principles for model-based prediction of human gait. J Biomech. 2010;43(6):1055–60.

Article
PubMed
PubMed Central
Google Scholar

Miller RH, Umberger BR, Hamill J, Caldwell GE. Evaluation of the minimum energy hypothesis and other potential optimality criteria for human running. Proc R Soc B. 2012;279(1733):1498–505.

Article
PubMed
Google Scholar

Song S, Geyer H. Predictive neuromechanical simulations indicate why walking performance declines with ageing. J Physiol. 2018;596(7):1199–210.

Article
CAS
PubMed
PubMed Central
Google Scholar

Chandler R, Clauser CE, McConville JT, Reynolds H, Young JW. Investigation of inertial properties of the human body. Air Force Aerospace Medical Research Lab Wright-Patterson AFB OH: Technical report; 1975.

Visser J, Hoogkamer J, Bobbert M, Huijing P. Length and moment arm of human leg muscles as a function of knee and hip-joint angles. Eur J Appl Physiol Occup Physiol. 1990;61(5–6):453–60.

Article
CAS
PubMed
Google Scholar

Yamaguchi G. A survey of human musculotendon actuator parameters. Multiple muscle systems: Biomechanics and movement organization; 1990.

Ward SR, Eng CM, Smallwood LH, Lieber RL. Are current measurements of lower extremity muscle architecture accurate? Clin Orthopaed Related Res. 2009;467(4):1074–82.

Article
Google Scholar

Scheys L, Loeckx D, Spaepen A, Suetens P, Jonkers I. Atlas-based non-rigid image registration to automatically define line-of-action muscle models: a validation study. J Biomech. 2009;42(5):565–72.

Article
PubMed
Google Scholar

Fregly BJ, Boninger ML, Reinkensmeyer DJ. Personalized neuromusculoskeletal modeling to improve treatment of mobility impairments: a perspective from european research sites. J Neuroeng Rehabil. 2012;9(1):18.

Article
PubMed
PubMed Central
Google Scholar

Kidziński Ł, Mohanty SP, Ong CF, Hicks JL, Carroll SF, Levine S, Salathé M, Delp SL. Learning to run challenge: Synthesizing physiologically accurate motion using deep reinforcement learning. In: The NIPS’17 Competition: Building Intelligent Systems, 2018;101–120. Springer.

SimTK: OpenSim. https://simtk.org/plugins/reports/index.php?type=group&group_id=91&reports=reports. Accessed 07 Aug 2021.

De Groote F, Van Campen A, Jonkers I, De Schutter J. Sensitivity of dynamic simulations of gait and dynamometer experiments to hill muscle model parameters of knee flexors and extensors. J Biomech. 2010;43(10):1876–83.

Article
PubMed
Google Scholar

Thelen DG, Anderson FC, Delp SL. Generating dynamic simulations of movement using computed muscle control. J Biomech. 2003;36(3):321–8.

Article
PubMed
Google Scholar

De Groote F, Kinney AL, Rao AV, Fregly BJ. Evaluation of direct collocation optimal control problem formulations for solving the muscle redundancy problem. Ann Biomed Eng. 2016;44(10):2922–36.

Article
PubMed
PubMed Central
Google Scholar

Liu MQ, Anderson FC, Schwartz MH, Delp SL. Muscle contributions to support and progression over a range of walking speeds. J Biomech. 2008;41(15):3243–52.

Article
PubMed
PubMed Central
Google Scholar

Hamner SR, Seth A, Delp SL. Muscle contributions to propulsion and support during running. J Biomech. 2010;43(14):2709–16.

Article
PubMed
PubMed Central
Google Scholar

Karabulut D, Dogru SC, Lin Y-C, Pandy MG, Herzog W, Arslan YZ. Direct validation of model-predicted muscle forces in the cat hindlimb during locomotion. J Biomech Eng. 2020;142:5.

Article
Google Scholar

Cavallaro EE, Rosen J, Perry JC, Burns S. Real-time myoprocessors for a neural controlled powered exoskeleton arm. IEEE Trans Biomed Eng. 2006;53(11):2387–96.

Article
PubMed
Google Scholar

Lotti N, Xiloyannis M, Durandau G, Galofaro E, Sanguineti V, Masia L, Sartori M. Adaptive model-based myoelectric control for a soft wearable arm exosuit: A new generation of wearable robot control. IEEE Robotics & Automation Magazine. 2020.

Uchida TK, Seth A, Pouya S, Dembia CL, Hicks JL, Delp SL. Simulating ideal assistive devices to reduce the metabolic cost of running. PLoS ONE. 2016;11:9.

Google Scholar

Fox MD, Reinbolt JA, Õunpuu S, Delp SL. Mechanisms of improved knee flexion after rectus femoris transfer surgery. J Biomech. 2009;42(5):614–9.

Article
PubMed
PubMed Central
Google Scholar

De Groote F, Falisse A. Perspective on musculoskeletal modelling and predictive simulations of human movement to assess the neuromechanics of gait. Proc R Soc B. 2021;288(1946):20202432.

Article
PubMed
PubMed Central
Google Scholar

Anderson FC, Pandy MG. Dynamic optimization of human walking. J Biomech Eng. 2001;123(5):381–90.

Article
CAS
PubMed
Google Scholar

Falisse A, Serrancolí G, Dembia CL, Gillis J, Jonkers I, De Groote F. Rapid predictive simulations with complex musculoskeletal models suggest that diverse healthy and pathological human gaits can emerge from similar control strategies. J R Soc Interface. 2019;16(157):20190402.

Article
PubMed
PubMed Central
Google Scholar

Miller RH, Umberger BR, Caldwell GE. Limitations to maximum sprinting speed imposed by muscle mechanical properties. J Biomech. 2012;45(6):1092–7.

Article
PubMed
Google Scholar

Handford ML, Srinivasan M. Energy-optimal human walking with feedback-controlled robotic prostheses: a computational study. IEEE Trans Neural Syst Rehabil Eng. 2018;26(9):1773–82.

Article
PubMed
Google Scholar

Zhang J, Fiers P, Witte KA, Jackson RW, Poggensee KL, Atkeson CG, Collins SH. Human-in-the-loop optimization of exoskeleton assistance during walking. Science. 2017;356(6344):1280–4.

Article
CAS
PubMed
Google Scholar

Thatte N, Geyer H. Toward balance recovery with leg prostheses using neuromuscular model control. IEEE Trans Biomed Eng. 2015;63(5):904–13.

Article
PubMed
PubMed Central
Google Scholar

Seo K, Hyung S, Choi, BK, Lee Y, Shim Y. A new adaptive frequency oscillator for gait assistance. In: 2015 IEEE International Conference on Robotics and Automation (ICRA), 2015;5565–71.

Batts Z, Song S, Geyer H. Toward a virtual neuromuscular control for robust walking in bipedal robots. In: 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2015;6318–23.

Van der Noot N, Ijspeert AJ, Ronsse R. Neuromuscular model achieving speed control and steering with a 3d bipedal walker. Auton Robots. 2019;43(6):1537–54.

Article
Google Scholar

Zhao G, Szymanski F, Seyfarth A. Bio-inspired neuromuscular reflex based hopping controller for a segmented robotic leg. Bioinspir Biomimet. 2020;15(2):026007.

Article
Google Scholar

Eilenberg MF, Geyer H, Herr H. Control of a powered ankle-foot prosthesis based on a neuromuscular model. IEEE Trans Neural Syst Rehabil Eng. 2010;18(2):164–73.

Article
PubMed
Google Scholar

Wu AR, Dzeladini F, Brug TJ, Tamburella F, Tagliamonte NL, Van Asseldonk EH, Van Der Kooij H, Ijspeert AJ. An adaptive neuromuscular controller for assistive lower-limb exoskeletons: a preliminary study on subjects with spinal cord injury. Front Neurorobot. 2017;11:30.

Article
PubMed
PubMed Central
Google Scholar

Orlovsky T, Orlovskiĭ GN, Deliagina T, Grillner S. Neuronal Control of Locomotion: from Mollusc to Man. Oxford: Oxford University Press; 1999.

Book
Google Scholar

Capaday C. The special nature of human walking and its neural control. TRENDS Neurosci. 2002;25(7):370–6.

Article
CAS
PubMed
Google Scholar

Armstrong DM. The supraspinal control of mammalian locomotion. J Physiol. 1988;405(1):1–37.

Article
CAS
PubMed
PubMed Central
Google Scholar

Sirota MG, Di Prisco GV, Dubuc R. Stimulation of the mesencephalic locomotor region elicits controlled swimming in semi-intact lampreys. Eur J Neurosci. 2000;12(11):4081–92.

Article
CAS
PubMed
Google Scholar

Clarac F. Some historical reflections on the neural control of locomotion. Brain Res Rev. 2008;57(1):13–21.

Article
PubMed
Google Scholar

Hultborn H. Spinal reflexes, mechanisms and concepts: from eccles to lundberg and beyond. Progr Neurobiol. 2006;78(3–5):215–32.

Article
Google Scholar

MacKay-Lyons M. Central pattern generation of locomotion: a review of the evidence. Phys Ther. 2002;82(1):69–83.

Article
PubMed
Google Scholar

Minassian K, Hofstoetter US, Dzeladini F, Guertin PA, Ijspeert A. The human central pattern generator for locomotion: Does it exist and contribute to walking? Neuroscientist. 2017;23(6):649–63.

Article
PubMed
Google Scholar

Lacquaniti F, Ivanenko YP, Zago M. Patterned control of human locomotion. J Physiol. 2012;590(10):2189–99.

Article
CAS
PubMed
PubMed Central
Google Scholar

Bizzi E, Cheung VC. The neural origin of muscle synergies. Front Comput Neurosci. 2013;7:51.

Article
PubMed
PubMed Central
Google Scholar

Ralston HJ. Energy-speed relation and optimal speed during level walking. Internationale Zeitschrift für Angewandte Physiologie Einschliesslich Arbeitsphysiologie. 1958;17(4):277–83.

CAS
PubMed
Google Scholar

Todorov E. Optimality principles in sensorimotor control. Nat Neurosci. 2004;7(9):907–15.

Article
CAS
PubMed
PubMed Central
Google Scholar

Matsuoka K. Sustained oscillations generated by mutually inhibiting neurons with adaptation. Biol Cybern. 1985;52(6):367–76.

Article
CAS
PubMed
Google Scholar

Taga G, Yamaguchi Y, Shimizu H. Self-organized control of bipedal locomotion by neural oscillators in unpredictable environment. Biol Cybern. 1991;65(3):147–59.

Article
CAS
PubMed
Google Scholar

Ogihara N, Yamazaki N. Generation of human bipedal locomotion by a bio-mimetic neuro-musculo-skeletal model. Biol Cybern. 2001;84(1):1–11.

Article
CAS
PubMed
Google Scholar

Hase K, Miyashita K, Ok S, Arakawa Y. Human gait simulation with a neuromusculoskeletal model and evolutionary computation. J Visualiz Comput Anim. 2003;14(2):73–92.

Article
Google Scholar

Jo S, Massaquoi SG. A model of cerebrocerebello-spinomuscular interaction in the sagittal control of human walking. Biol Cybern. 2007;96(3):279–307.

Article
PubMed
Google Scholar

Aoi S, Ohashi T, Bamba R, Fujiki S, Tamura D, Funato T, Senda K, Ivanenko Y, Tsuchiya K. Neuromusculoskeletal model that walks and runs across a speed range with a few motor control parameter changes based on the muscle synergy hypothesis. Sci Rep. 2019;9(1):1–13.

Article
CAS
Google Scholar

Günther M, Ruder H. Synthesis of two-dimensional human walking: a test of the λ-model. Biol Cybern. 2003;89(2):89–106.

Article
PubMed
Google Scholar

Dzeladini F, Van Den Kieboom J, Ijspeert A. The contribution of a central pattern generator in a reflex-based neuromuscular model. Front Human Neurosci. 2014;8:371.

Article
Google Scholar

Wang J, Qin W, Sun L. Terrain adaptive walking of biped neuromuscular virtual human using deep reinforcement learning. IEEE Access. 2019;7:92465–75.

Article
Google Scholar

Song S, Geyer H. Evaluation of a neuromechanical walking control model using disturbance experiments. Front Comput Neurosci. 2017;11:15.

Article
PubMed
PubMed Central
Google Scholar

Faloutsos P, Van de Panne M, Terzopoulos D. Composable controllers for physics-based character animation. In: Proceedings of the 28th ACM SIGGRAPH, 2001;251–60.

Fang AC, Pollard NS. Efficient synthesis of physically valid human motion. ACM Trans Graph. 2003;22(3):417–26.

Article
Google Scholar

Wampler K, Popović Z. Optimal gait and form for animal locomotion. ACM Trans Graph. 2009;28(3):1–8.

Article
Google Scholar

Coros S, Karpathy A, Jones B, Reveret L, Van De Panne M. Locomotion skills for simulated quadrupeds. ACM Trans Graph. 2011;30(4):1–12.

Article
Google Scholar

Levine S, Popović J. Physically plausible simulation for character animation. In: Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation,2012;221–230. Eurographics Association.

Yin K, Loken K, Van de Panne M. Simbicon: Simple biped locomotion control. ACM Trans Graph. 2007;26(3):105.

Article
CAS
Google Scholar

Wu C-C, Zordan V. Goal-directed stepping with momentum control. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, 2010;113–8.

De Lasa M, Mordatch I, Hertzmann A. Feature-based locomotion controllers. ACM Trans Graph. 2010;29(4):1–10.

Article
Google Scholar

Coros S, Beaudoin P, Van de Panne M. Generalized biped walking control. ACM Trans Graph. 2010;29(4):1–9.

Article
Google Scholar

Zordan VB, Hodgins JK. Motion capture-driven simulations that hit and react. In: Proceedings of the 2002 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, 2002;89–96.

Da Silva M, Abe Y, Popović J. Simulation of human motion data using short-horizon model-predictive control. In: Computer Graphics Forum, 2008;27, 371–380. Wiley Online Library.

Lee Y, Kim S, Lee J. Data-driven biped control. In: ACM SIGGRAPH 2010 Papers, 2010;1–8.

Hong S, Han D, Cho K, Shin JS, Noh J. Physics-based full-body soccer motion control for dribbling and shooting. ACM Trans Graph. 2019;38(4):1–12.

Article
Google Scholar

Wang JM, Hamner SR, Delp SL, Koltun V. Optimizing locomotion controllers using biologically-based actuators and objectives. ACM Trans Graph. 2012;31(4):1–11.

Article
Google Scholar

Geijtenbeek T, Van De Panne M, Van Der Stappen AF. Flexible muscle-based locomotion for bipedal creatures. ACM Trans Graph. 2013;32(6):1–11.

Article
Google Scholar

Lee Y, Park MS, Kwon T, Lee J. Locomotion control for many-muscle humanoids. ACM Trans Graph. 2014;33(6):1–11.

Article
Google Scholar

Kuo AD. Stabilization of lateral motion in passive dynamic walking. Int J Robot Res. 1999;18(9):917–30.

Article
Google Scholar

Obinata G, Hase K, Nakayama A. Controller design of musculoskeletal model for simulating bipedal walking. In: Annual Conference of the International FES Society, 2004;2, p. 1.

Song S, Collins SH. Optimizing exoskeleton assistance for faster self-selected walking. IEEE Trans Neural Syst Rehabil Eng. 2021;29:786–95.

Article
PubMed
PubMed Central
Google Scholar

Choi JT, Bastian AJ. Adaptation reveals independent control networks for human walking. Nat Neurosci. 2007;10(8):1055–62.

Article
CAS
PubMed
Google Scholar

Torres-Oviedo G, Bastian AJ. Natural error patterns enable transfer of motor learning to novel contexts. J Neurophysiol. 2012;107(1):346–56.

Article
PubMed
Google Scholar

Emken JL, Benitez R, Sideris A, Bobrow JE, Reinkensmeyer DJ. Motor adaptation as a greedy optimization of error and effort. J Neurophysiol. 2007;97(6):3997–4006.

Article
PubMed
Google Scholar

Cajigas I, Koenig A, Severini G, Smith M, Bonato P. Robot-induced perturbations of human walking reveal a selective generation of motor adaptation. Sci Robot. 2017;2(6):1–10.

Article
Google Scholar

Sutton RS, Barto AG. Reinforcement Learning: An Introduction. MIT press; 2018.

Levine S. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv:1805.00909, 2018.

Schulman J, Heess N, Weber T, Abbeel P. Gradient estimation using stochastic computation graphs. arXiv preprint arXiv:1506.05254,2015.

Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, Riedmiller M, Fidjeland AK, Ostrovski G, et al. Human-level control through deep reinforcement learning. Nature. 2015;518(7540):529–33.

Article
CAS
PubMed
Google Scholar

Silver D, Huang A, Maddison CJ, Guez A, Sifre L, Van Den Driessche G, Schrittwieser J, Antonoglou I, Panneershelvam V, Lanctot M, et al. Mastering the game of go with deep neural networks and tree search. Nature. 2016;529(7587), 484.

Silver D, Schrittwieser J, Simonyan K, Antonoglou I, Huang A, Guez A, Hubert T, Baker L, Lai M, Bolton A, et al. Mastering the game of go without human knowledge. Nature. 2017;550(7676):354–9.

Article
CAS
PubMed
Google Scholar

Vinyals O, Babuschkin I, Czarnecki WM, Mathieu M, Dudzik A, Chung J, Choi DH, Powell R, Ewalds T, Georgiev P, et al. Grandmaster level in Starcraft II using multi-agent reinforcement learning. Nature. 2019;575(7782):350–4.

Article
CAS
PubMed
Google Scholar

OpenAI Spinning Up Documentation. https://spinningup.openai.com/en/latest/spinningup/rl_intro2.html#id20. Accessed 08 Aug 2020.

Schulman J, Levine S, Abbeel P, Jordan M, Moritz P. Trust region policy optimization. In: International Conference on Machine Learning, 2015;1889–97.

Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 ,2017.

Lillicrap TP, Hunt JJ, Pritzel A, Heess N, Erez T, Tassa Y, Silver D, Wierstra D. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.

Fujimoto S, Van Hoof H, Meger D. Addressing function approximation error in actor-critic methods. In: International Conference on Machine Learning, 2018, p. 1587–96.

Haarnoja T, Zhou A, Abbeel P, Levine S. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In: International Conference on Machine Learning; 2018, p. 1861–70.

Sutton RS, McAllester DA, Singh SP, Mansour Y. Policy gradient methods for reinforcement learning with function approximation. In: Advances in Neural Information Processing Systems, 2000;1057–63.

Kakade S, Langford J. Approximately optimal approximate reinforcement learning. In: International Conference on Machine Learning, vol. 2. 2002;267–74.

Deisenroth MP, Neumann G, Peters J, et al. A survey on policy search for robotics. Foundations and Trends in Robotics 2013;2(1–2), 1–142.

Hansen N. The cma evolution strategy: a comparing review. In: Towards a New Evolutionary Computation, pp. 75–102. Springer,2006.

Wang JM, Fleet DJ, Hertzmann A. Optimizing walking controllers for uncertain inputs and environments. ACM Trans Graph. 2010;29(4):1–8.

Article
Google Scholar

Hansen N, Müller SD, Koumoutsakos P. Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (cma-es). Evol Comput. 2003;11(1):1–18.

Article
PubMed
Google Scholar

Peng XB, Berseth G, Van de Panne M. Terrain-adaptive locomotion skills using deep reinforcement learning. ACM Trans Graph. 2016;35(4):1–12.

Article
CAS
Google Scholar

Yu W, Turk G, Liu CK. Learning symmetric and low-energy locomotion. ACM Trans Graph. 2018;37(4):1–12.

Article
Google Scholar

Liu L, Hodgins J. Learning basketball dribbling skills using trajectory optimization and deep reinforcement learning. ACM Trans Graph. 2018;37(4):1–14.

Article
Google Scholar

Clegg A, Yu W, Tan J, Liu CK, Turk G. Learning to dress: Synthesizing human dressing motion via deep reinforcement learning. ACM Trans Graph. 2018;37(6):1–10.

Article
Google Scholar

Peng XB, Berseth G, Yin K, Van De Panne M. Deeploco: Dynamic locomotion skills using hierarchical deep reinforcement learning. ACM Trans Graph. 2017;36(4):1–13.

Article
Google Scholar

Heess N, TB D, Sriram S, Lemmon J, Merel J, Wayne G, Tassa Y, Erez T, Wang Z, Eslami S, Riedmiller M, Silver David. Emergence of locomotion behaviours in rich environments. arXiv preprint arXiv:1707.02286, 2017.

Peng XB, van de Panne M. Learning locomotion skills using deeprl: Does the choice of action space matter? In: Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation, 2017;1–13.

Brockman G, Cheung V, Pettersson L, Schneider J, Schulman J, Tang J, Zaremba W. Openai gym. arXiv preprint arXiv:1606.01540, 2016.

OpenAI Spinning Up. https://anaconda.org/kidzik/opensim. Accessed 08 Aug 2020.

Kidziński Ł , Mohanty SP, Ong CF, Huang Z, Zhou S, Pechenko A, Stelmaszczyk A, Jarosik P, Pavlov M, Kolesnikov S, et al. Learning to run challenge solutions: Adapting reinforcement learning methods for neuromusculoskeletal environments. In: The NIPS’17 Competition: Building Intelligent Systems, 2018;121–153. Springer.

Kidziński Ł, Ong C, Mohanty SP, Hicks J, Carroll S, Zhou B, Zeng H, Wang F, Lian R, Tian H, et al. Artificial intelligence for prosthetics: Challenge solutions. In: The NeurIPS’18 Competition, 2020;69–128. Springer.

osim-rl. https://github.com/stanfordnmbl/osim-rl/tree/v3.0.11. Accessed 08 Aug 2020.

AIcrowd —- Challenges - NeurIPS 2019: Learn to Move. https://www.aicrowd.com/challenges/neurips-2019-learning-to-move-walk-around. Accessed 08 Aug 2020.

osim-rl. http://osim-rl.stanford.edu/. Accessed 08 Aug 2020.

Schwartz MH, Rozumalski A, Trost JP. The effect of walking speed on the gait of typically developing children. J Biomech. 2008;41(8):1639–50.

Article
PubMed
Google Scholar

Hamner SR, Delp SL. Muscle contributions to fore-aft and vertical body mass center accelerations over a range of running speeds. J Biomech. 2013;46(4):780–7.

Article
PubMed
Google Scholar

John CT, Anderson FC, Higginson JS, Delp SL. Stabilisation of walking by intrinsic muscle properties revealed in a three-dimensional muscle-driven simulation. Comput Methods Biomech Biomed Eng. 2013;16(4):451–62.

Article
Google Scholar

Zhou B, Zeng H, Wang F, Li Y, Tian H. Efficient and robust reinforcement learning with uncertainty-based value expansion. arXiv preprint arXiv:1912.05328, 2019.

Kolesnikov S, Hrinchuk O. Sample efficient ensemble learning with catalyst.rl. arXiv preprint arXiv:2003.14210, 2020

Akimov D. Distributed soft actor-critic with multivariate reward representation and knowledge distillation. arXiv preprint arXiv:1911.13056, 2019.

Bengio Y, Louradour J, Collobert R, Weston J. Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning, 2009;41–48.

Hase K, Stein R. Turning strategies during human walking. J Neurophysiol. 1999;81(6):2914–22.

Article
CAS
PubMed
Google Scholar

Taylor MJD, Dabnichki P, Strike S. A three-dimensional biomechanical comparison between turning strategies during the stance phase of walking. Human Movement Sci. 2005;24(4):558–73.

Article
CAS
Google Scholar

Hallemans A, De Clercq D, Aerts P. Changes in 3d joint dynamics during the first 5 months after the onset of independent walking: a longitudinal follow-up study. Gait Posture. 2006;24(3):270–9.

Article
PubMed
Google Scholar

Adolph KE, Cole WG, Komati M, Garciaguirre JS, Badaly D, Lingeman JM, Chan GL, Sotsky RB. How do you learn to walk? thousands of steps and dozens of falls per day. Psychol Sci. 2012;23(11):1387–94.

Article
PubMed
Google Scholar

Heess N, Wayne G, Tassa Y, Lillicrap T, Riedmiller M, Silver D. Learning and transfer of modulated locomotor controllers. arXiv preprint arXiv:1610.05182, 2016.

Frans K, Ho J, Chen X, Abbeel P, Schulman J. Meta learning shared hierarchies. arXiv preprint arXiv:1710.09767, 2017.

Merel J, Botvinick M, Wayne G. Hierarchical motor control in mammals and machines. Nat Commun. 2019;10(1):1–12.

Article
CAS
Google Scholar

Latombe J-C. Robot Motion Planning, vol. 124. Springer; 2012.

Paden B, Čáp M, Yong SZ, Yershov D, Frazzoli E. A survey of motion planning and control techniques for self-driving urban vehicles. IEEE Trans Intell Vehicles. 2016;1(1):33–55.

Article
Google Scholar

World Chase Tag\(^{\rm TM}\) - Home. https://www.worldchasetag.com/. Accessed 08 Aug 2020.

Afschrift M, De Groote F, Jonkers I. Similar sensorimotor transformations control balance during standing and walking. PLoS Comput Biol. 2021;17(6).

Müller R, Vielemeyer J, Häufle DF. Negotiating ground level perturbations in walking: Visual perception and expectation of curb height modulate muscle activity. J Biomech. 2020;113:110121.

Article
PubMed
Google Scholar

Chambers V, Artemiadis P. A model-based analysis of supraspinal mechanisms of inter-leg coordination in human gait: Toward model-informed robot-assisted rehabilitation. IEEE Trans Neural Syst Rehab Eng. 2021;29:740–9.

Article
Google Scholar

Sánchez N, Simha SN, Donelan JM, Finley JM. Using asymmetry to your advantage: learning to acquire and accept external assistance during prolonged split-belt walking. J Neurophysiol. 2021;125(2):344–57.

Article
PubMed
Google Scholar

Smith MA, Ghazizadeh A, Shadmehr R. Interacting adaptive processes with different timescales underlie short-term motor learning. PLoS Biol. 2006;4:6.

Article
CAS
Google Scholar

Caporale N, Dan Y. Spike timing-dependent plasticity: a hebbian learning rule. Annu Rev Neurosci. 2008;31:25–46.

Article
CAS
PubMed
Google Scholar

Shadmehr R, Smith MA, Krakauer JW. Error correction, sensory prediction, and adaptation in motor control. Ann Rev Neurosci. 2010;33:89–108.

Article
CAS
PubMed
Google Scholar

Wolpert DM, Diedrichsen J, Flanagan JR. Principles of sensorimotor learning. Nat Rev Neurosci. 2011;12(12):739–51.

Article
CAS
PubMed
Google Scholar

Haith AM, Krakauer JW. Theoretical models of motor control and motor learning. In: Routledge Handbook of Motor Control and Motor Learning, 2013;16–37. Routledge.

Tedrake R, Zhang TW, Seung HS. Stochastic policy gradient reinforcement learning on a simple 3d biped. In: 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)(IEEE Cat. No. 04CH37566), 2004;3, 2849–54.

Manoonpong P, Geng T, Kulvicius T, Porr B, Wörgötter F. Adaptive, fast walking in a biped robot under neuronal control and learning. PLoS Comput Biol. 2007;3(7):134.

Article
Google Scholar

Sar P, Geyer H. A model for the transfer of control from the brain to the spinal cord through synaptic learning. J Comput Neurosci. 2020;48(4):365–75.

Article
PubMed
Google Scholar

Alexander RM. Simple models of human movement,1995.

Seyfarth A, Geyer H, Günther M, Blickhan R. A movement criterion for running. J Biomech. 2002;35(5):649–55.

Article
PubMed
Google Scholar

Geyer H, Seyfarth A, Blickhan R. Compliant leg behaviour explains basic dynamics of walking and running. Proc R Soc Lond B. 2006;273(1603):2861–7.

Google Scholar

Srinivasan M, Ruina A. Computer optimization of a minimal biped model discovers walking and running. Nature. 2006;439(7072):72–5.

Article
CAS
PubMed
Google Scholar

Bauby CE, Kuo AD. Active control of lateral balance in human walking. J Biomech. 2000;33(11):1433–40.

Article
CAS
PubMed
Google Scholar

Bertram JE, Ruina A. Multiple walking speed-frequency relations are predicted by constrained optimization. J Theor Biol. 2001;209(4):445–53.

Article
CAS
PubMed
Google Scholar

Arechavaleta G, Laumond J-P, Hicheur H, Berthoz A. An optimality principle governing human walking. IEEE Trans Robot. 2008;24(1):5–14.

Article
Google Scholar

Brown GL, Seethapathi N, Srinivasan M. Energy optimality predicts curvilinear locomotion. arXiv preprint arXiv:2001.02287, 2020.

Srinivasan M. Fifteen observations on the structure of energy-minimizing gaits in many simple biped models. J R Soc Interface. 2011;8(54):74–98.

Article
PubMed
Google Scholar

Faraji S, Wu AR, Ijspeert AJ. A simple model of mechanical effects to estimate metabolic cost of human walking. Sci Rep. 2018;8(1):1–12.

Google Scholar

Darici O, Kuo AD. Humans optimally anticipate and compensate for an uneven step during walking. bioRxiv, 2021;2020–12.

Kuo AD. The relative roles of feedforward and feedback in the control of rhythmic movements. Motor control. 2002;6(2):129–45.

Article
PubMed
Google Scholar

Geyer H, Seyfarth A, Blickhan R. Positive force feedback in bouncing gaits? Proc R Soc London B. 2003;270(1529):2173–83.

Article
Google Scholar

Sanchez N, Simha SN, Donelan JM, Finley JM. Taking advantage of external mechanical work to reduce metabolic cost: the mechanics and energetics of split-belt treadmill walking. J Physiol. 2019;597(15):4053–68.

Article
CAS
PubMed
Google Scholar

Seethapathi N, Clark B, Srinivasan M. Exploration-based learning of a step to step controller predicts locomotor adaptation. bioRxiv, 2021.

Winter DA. Biomechanics and Motor Control of Human Movement. New York: Wiley; 2009.

Book
Google Scholar