The problem with a conversation
An unstructured interview reliably tells you who is comfortable in an interview. That is a real skill, but it is rarely the skill you are hiring for, and it correlates poorly with performance in the job. Confidence gets read as competence, and rapport gets read as fit.
Agree the scorecard before you meet anyone
Write down the four or five things the person must be able to do, and how you would recognise each. Do it before the first interview, or the criteria will quietly reshape themselves around whoever you happened to like most.
Ask everyone the same questions
The same core questions, in the same order, scored independently before the panel discusses anyone. Comparability is the entire point — without it you are comparing impressions formed in different conversations.
Ask about what they did, not what they would do
Hypothetical questions measure how well someone imagines. Ask instead for a specific instance: what the situation was, what they personally did, and how it turned out. Then follow up on the detail — that is where the difference between having done something and having watched it happen becomes obvious.
Score before you discuss
If the panel debriefs before scoring, the most senior or most confident voice anchors everyone else. Independent scores first, discussion second, and the disagreements become the most useful part of the conversation.