Debate over whether distillation constitutes theft or legitimate technique, comparison to how GPT trained on internet data, noting API fees were paid, irony of Anthropic criticizing distillation while pirating books for training data
The debate over model distillation reveals a sharp divide between those who view it as "theft" and those who see it as a natural, recursive progression of technology akin to how GPT distilled the internet. Commenters highlight the irony of major AI labs condemning the practice while facing their own legal battles over pirated training data, noting that distillation at least involves paying for API access. While some argue that relying on distillation prevents a developer from ever reaching the true technological frontier, others defend it as a pragmatic strategy for efficiency, comparing it to the common use of generic brands in other industries.
7 comments tagged with this topic