What Are Realistic GPT-4 Size Expectations?

This article aims to cut through the hype by considering historic scaling and current trends of existing LLM models.

I say this for 3️⃣ reasons…

⭐️ Please follow me on LinkedIn for updates on Conversational AI ⭐️

1️⃣ The LLM Trends point to a one trillion model, or close to it.

A billion = 1,000,000,000 & a trillion = 1,000,000,000,000

GPT-1           117 000 000
GPT-Neo 1 300 000 000
GPT-2 1 500 000 000
GPT-J 6 000 000 000
Fairseq 13 000 000 000
GPT-NeoX 20 000 000 000
Chinchilla 70 000 000 000
LaMDA 137 000 000 000
GTP-3 175 000 000 000
BLOOM 176 000 000 000
Gopher 280 000 000 000
MT-NLG 530 000 000 000
PaLM 540 000 000 000
GPT-4 1 000 000 000 000

2️⃣ The Experts point to a scenario of less parameters, not more

I recently posed the question on LinkedIn, regarding the GPT-4 parameter count.

3️⃣ There is only so much quality data

A recent study found that stock of high-quality language data will be exhausted in all likelihood by 2026. In contrast, low-quality language and image data will be exhausted only between 2030 and 2050.

Language datasets have grown exponentially by more than 50% per year, and contain up to 2e12 words as of October 2022.

The stock of language data currently grows by 7% yearly, but our model predicts a slowdown to 1% by 2100.

In Conclusion

There are two elements which capture the imagination of the general public when it comes to AI.

https://www.linkedin.com/in/cobusgreyling
https://www.linkedin.com/in/cobusgreyling

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Cobus Greyling

Chief Evangelist @ HumanFirst. I explore and write about all things at the intersection of AI and language; NLP/NLU/LLM, Chat/Voicebots, CCAI. www.humanfirst.ai