Title
Deep Generative Models that Solve PDEs: Distributed Computing for Training Large Data-Free Models
Campus Units
Mechanical Engineering, Electrical and Computer Engineering, Plant Sciences Institute
Document Type
Article
Publication Version
Submitted Manuscript
Publication Date
7-24-2020
Journal or Book Title
arXiv
Abstract
Recent progress in scientific machine learning (SciML) has opened up the possibility of training novel neural network architectures that solve complex partial differential equations (PDEs). Several (nearly data free) approaches have been recently reported that successfully solve PDEs, with examples including deep feed forward networks, generative networks, and deep encoder-decoder networks. However, practical adoption of these approaches is limited by the difficulty in training these models, especially to make predictions at large output resolutions (≥1024×1024). Here we report on a software framework for data parallel distributed deep learning that resolves the twin challenges of training these large SciML models - training in reasonable time as well as distributing the storage requirements. Our framework provides several out of the box functionality including (a) loss integrity independent of number of processes, (b) synchronized batch normalization, and (c) distributed higher-order optimization methods. We show excellent scalability of this framework on both cloud as well as HPC clusters, and report on the interplay between bandwidth, network topology and bare metal vs cloud. We deploy this approach to train generative models of sizes hitherto not possible, showing that neural PDE solvers can be viably trained for practical applications. We also demonstrate that distributed higher-order optimization methods are 2−3× faster than stochastic gradient-based methods and provide minimal convergence drift with higher batch-size.
Copyright Owner
The Author(s)
Copyright Date
2020
Language
en
File Format
application/pdf
Recommended Citation
Botelho, Sergio; Joshi, Ameya; Khara, Biswajit; Sarkar, Soumik; Hegde, Chinmay; Adavani, Santi; and Ganapathysubramanian, Baskar, "Deep Generative Models that Solve PDEs: Distributed Computing for Training Large Data-Free Models" (2020). Mechanical Engineering Publications. 429.
https://lib.dr.iastate.edu/me_pubs/429
Comments
This is a pre-print of the article Botelho, Sergio, Ameya Joshi, Biswajit Khara, Soumik Sarkar, Chinmay Hegde, Santi Adavani, and Baskar Ganapathysubramanian. "Deep Generative Models that Solve PDEs: Distributed Computing for Training Large Data-Free Models." arXiv preprint arXiv:2007.12792 (2020). Posted with permission.