Archives for The Pile dataset training
10
Feb
Microsoft, NVIDIA test waters for a large-scale generative language model with promising results
We believe that our results and findings can help, shape, and facilitate future research in foundational, large-scale pretraining.
The key is that 1T was never ‘trained to convergence.’
MT-NLG has 3x the number of parameters compared to the existing largest models – GPT-3, Turing NLG, Megatron-LM and others.