As a minor example, consider the
popcount() code example from https://www.pcmag.com/news/samsung-software-engineers-busted-for-pasting-proprietary-code-into-chatgpt.
When asked to correct the following code ChatGPT claims the fix is cleaning up some non-ascii characters and claims the code computes the number of bits set in the binary representation of the integer
n (call this ideal quantity
However, believing that depends on not thinking about the code in terms of invariants. For the above code to work we would need to have the invariant that
count + popcount(n) is a constant as we move through the loop. This would require the invariant that
n ^ (n - 1) has one fewer bit set in the base-2 representation than
count increases by this much).
None of that is the case.
Consider working the example
n = 1.
n = 1 n ^ (n - 1)
1, meaning the function will cycle- never returning any value.
It appears some fraction of the magic of ChatGPT answers depends on not caring enough to read the answers carefully. The AIs work well in a world where nobody cares about the work. This is part of why they will in fact dominate writing tasks: nobody reads carefully.
It isn’t just the AI’s that are “hallucinating.” Some of what they do is to form a text-mirror where the scorer is impressed by the training data and image of their own actions.
This sort of garbage in garbage out nonsense is already infecting search engine context expansion: