It is a partial analogy, it takes into consideration the outputs which are related to some specific training data and disconsiders the outputs which cannot be directly related to any specific training data.
For example, make up a new meme template and a new joke on the spot, it couldn’t have seen it before if you make sure your joke and template are new. If the AI can explain it then compression is a horrendous analogy.
Lossy compression explains outputs being similar but not identical when trying to recover the original data, it doesn’t explain brand new content that makes sense standalone. Imagine a lossy audio compression resulting in a brand new song midway through playback, or a lossy image compression resulting in a brand new coherent image being overlayed onto some pixels of the original image. That is not what happens, lossy audio compression results in noise, lossy image compression results in noise, not in coherent unheard songs and unseen images.
TBF, compression is related to ML. Hence, the Hutter Prize. Thinking of LLMs as lossy compression algorithms is a decent analogy.
It is a partial analogy, it takes into consideration the outputs which are related to some specific training data and disconsiders the outputs which cannot be directly related to any specific training data.
For example, make up a new meme template and a new joke on the spot, it couldn’t have seen it before if you make sure your joke and template are new. If the AI can explain it then compression is a horrendous analogy.
Lossy compression explains outputs being similar but not identical when trying to recover the original data, it doesn’t explain brand new content that makes sense standalone. Imagine a lossy audio compression resulting in a brand new song midway through playback, or a lossy image compression resulting in a brand new coherent image being overlayed onto some pixels of the original image. That is not what happens, lossy audio compression results in noise, lossy image compression results in noise, not in coherent unheard songs and unseen images.