LLMs1 min read
Looped GPT-BERT Achieves Comparable Performance with Fewer Parameters
Researchers demonstrated Looped GPT-BERT, a model utilizing depth-wise parameter sharing and recurrent traversals, achieving performance comparable to larger GPT-2 and GPT-BERT models on benchmarks like BLiMP and GLUE. The design, trained on a 7.48M-word corpus, offers a more efficient approach to language modeling.
From arXiv cs.CL