Gshard

GShard is a intra-layer parallel distributed method. It consists of set of simple APIs for annotations, and a compiler extension in XLA for automatic parallelization.GShard is a deep learning framework developed by Google Research for large-scale, distributed training of language models

Resources

• GShard
• The research paper introducing GShard
• TensorFlow documentation for GShard
• A blog post by Google Research introducing GShard
• the concept of GShard in machine learning