I interviewed a candidate recently where everyone on the panel is convinced the person was reading the output of an LLM in real time. It seems likely that they had the audio piped to some kind of system that heard the questions and wrote plausible replies. Of course none of us KNOW this to be true, we just strongly suspect it.
They would say things like "we implemented a privacy protocol" and I would say "I'm in security, but privacy really isn't my area. Can you explain a bit what a 'privacy protocol' is?" You know the thing they JUST said in the immediately prior sentence? And the candidate was stumped. They hemmed and hawwed and delayed with phrases like "mmm, let me think of just the right way to answer that..."
It is totally trivial to detect and defeat someone interviewing this way. All you have to do is ask follow-up questions. "That's interesting, what did you do next?" or "Tell me how you decided that was the right thing to do?" LLMs don't tell coherent stories. They're just making stuff up. All you have to do is ask for details. The details won't tell a consistent story. They told me about a situation that was a "suspected data breach" but it turned out to be a false alarm. I asked "what gave you confidence that it was not a data breach?" and they really struggled to answer that.
The other thing that was laughable was their approach to the code and design exercise. Given a problem description, they were able to—almost instantly—verbally outline the right solution. And when asked questions like "what's the computational complexity?" they could provide the right answer (e.g., "O(n)" or "O(log n)"). And then you ask a really simple follow-up question like "so what's the outermost for() loop going to look like?" and they can't answer. There is no for() loop in their head.
They didn't want to be obvious in their copy/pasting of code from an LLM, so they typed. But since they didn't understand actual Python syntax, they used the wrong quote marks (e.g., ` instead of ') and they didn't take care with indentation.
I'm told this is a problem in lots of roles at lots of companies. Heaven help us all.
#employment #interview #jobinterviews #jobhunt