The rise of LLMs questions our common understanding of "size." The infinitely large—billions of parameters as a measure of the immeasurable—meets the infinitely small: high-performance chips with nanoscopic engravings. As Pierre-Carl Langlais rightly points out, the notion of size becomes entirely relative in the realm of large language models. "Large," yes, but compared to what?
I like to think that history and literature can shed light on the technological revolutions we are experiencing, when we lack perspective. For example, the character of Gargantua, depicted in the sixteenth century in Rabelais' novel, is an interesting parallel with this technological moment we are experiencing: just like Gargantua was born enormous and hungry—of both food and knowledge, by the way—so is the need for training material of large models like GPT-4, which have "swallowed" immense quantities of data, the legal implications of which are only beginning to be understood, as the emerging disputes with major publishers and video platforms allegedly exploited make clear. The motto from Gargantua's utopia, "Fais ce que voudras" (Do what thou wilt), also mirrors the dual-edged sword of LLM development, where little to no legal framework exists, while in the meantime investor expectations push the boundaries of ethical and legal considerations.
Just as Gargantua eventually grows from a voracious giant to a figure of Renaissance wisdom, so must our approach to LLMs evolve. Fortunately, the open-source ecosystem now seems to favor more granular, composable, and therefore more frugal architectures.
