Hallucinations as a term for incorrect answers made by token generators is a genius marketing term. In fact, labelling LLMs as AI is what propelled high valuations of these companies.
This is what leads to people even in the industry to take at face value a human-like response in helping answer a query; and in this case, willingly downloading malware.
I'm just waiting for AI to suggest to download package X for usage in language A, which exists in language B but not A, and some malicious actor creating X in A to spread their malware.
In the original Llama paper, the process of preparing the corpus was described. For the sourcecode aspect, it was fed into their model's corpus after being cleaned of boilerplate code. I think it'd be a fair assumption that most of the other vendors followed this practice when creating their datasets.
I'm not a python user, but in most languages, libraries are referenced in (what most devs would consider) boilerplate code. Purely conjecture, but perhaps without boilerplate code, the LLMs are left guessing the names of popular libraries and just merges together two common naming conventions "huggingface" and "-cli".
It used to be garbage in, garbage out (GIGO). But now, sometimes you put valid data in and get garbage out. I just can't go all-in on LLMs with the error / hallucinate rate where it currently is. And people say it's getting better. But I guess I'll just do things the slower, and more accurate way until such time arrives.
A nice thing to do might be to put "Mountweazel" packages under names like this, i.e. a package that always fails to install and gives an error explaining you've been duped by LLMinati.
There is already something like this for e.g. `nvidia-tlt` which exists on PyPI, but just as a placeholder telling you to go and add Nvidia's pip repository.
I get the malware-infection potential, but it's still blows my mind that the fake package wasn't caught earlier by the actual users. All those downloads and nobody wondered why X feature that depends on it wasn't working? Whatever happened to testing things and having decent coverage?
An LLM gets things wrong. It's no different than interacting with a person. You will get a confident answer. The difference is you can easily get a second opinion of an LLM without offending it. Hallucinations are unlikely to occur twice in a row. So just ask twice.
People will say this happened before with people purposefully doing this, but it will happen at a much faster rate with much less oversight with AI. Moreover, I can't help but think of the comments both Garry Kasparov and Go players made when they were playing against AI: they thought it had very counterintuitive and confusing moves compared to human play.
The truth is, while both humans and AI can make errors, and both can be malicious as well, the actions of AI will be counterintuitive and confusing and we won't know how to counter them in the same way that we counter human follies.
This is just one aspect that shows that AI is making society worse on average and it should be destroyed.
copilot hallucinates functions and packages versions all the time, it's a nightmare, I never know if I got a suggestion from a predictible IDE algorithm or the drunk overeager intern, and it's really annoying. Being on your toes means you have no mental flow, because the pothead intern is unpredictible in its suggestions.
Some of us here may be old enough to remember the days when Microsoft was pushing ActiveX controls as the future of the web. Basically they were binary components that would run in your browser and weren't really effectively sandboxed. Those of us that pointed out this was a major security hole were ignored. Inevitably carnage followed.
People blindly accepting the output of LLMs seems similarly crazy to me and I think it's only a matter of time before we face a real reckoning over this. The lesson here I think is just because a lot of people are advocating something that seems reckless doesn't mean it isn't reckless.
https://www.howtogeek.com/162282/what-activex-controls-are-a...