A big issue when it comes to LLMS discussions these days, is figuring out when an AI should be allowed to make judgements about the real world. For example, suppose gpt-4 is trained on some medical research papers, when would it be justified to pass medical diagnoses like a doctor? Or, before that even, should we take any judgement on a serious matter by an LLM seriously for that matter?
This genAI post suggests that chatgpt is already good at forming such hypotheses based on research paper.
Right now there is no so-called "artificial intelligence" that could be trusted to make such a judgement. An LLM is trained on data; later it can find a sentence or sentences that match up with the input is given, but it doesn't understand the training data. It doesn't actually understand what exactly in the training data implies which illness. It will create a model of what it thinks the correlation is, but that correlation can be completely wrong.
So you wait at the very least until you have evidence that such things don't happen. What you can do is give a suggested diagnosis, or two suggestions, and let a qualified doctor decide what the actual diagnosis is. But your actual question... We'll have to wait until the situation actually arises.
So there's a counterpart to all of the development in large language modelling, and that's the construction of Knowledge Representation systems.
A core idea in ethical AI is that the accountable judgements are not really made in the course of the language processing model deciding how to respond to your prompt, but rather that what the chat agent does is direct the user towards the appropriate parts of the represented knowledge (properly cited or tagged), and that the ontology/represented knowledge is where the questions of reliability and accountability are answered. The AI is basically just the call centre agent - the script was written by the human team behind the scenes, and it's the team, rather than the chatbot, that does the hard work of grounding the symbols in facts.
ChatGPT might, in principle, be able to reference public ontologies and semantic frameworks quite well without needing to be connected to any given individual ontology, but what's needed to make this work is our AI will need to have the capacity to demonstrate good information literacy practices as standard, and moreover people will need to keep that practice accountable. Something like a Core AI international working group might be a useful outcome of this, similar to e.g. ICANN for the internet.
This reminds me a lot of AI-based self-driving cars, which have gained quite a lot of trust already, but it also still has a long way to go. AI also makes decisions in many other places (e.g. fraud detection). I'd say ChatGPT is far behind that in terms of trust in anything resembling a regulated environment, but it might show a possible path of where such models might go in future, if trust increases.
Justifiable trust in a method or source of truth seems to come from 2 places:
If following the same method consistently produces results we've verified as true, this would increase trust.
ChatGPT says a lot that's right, but has frequently shown to just make stuff up, so that's not particularly reliable.
You could also question the types of reliability. If someone comes in with a common cold, you don't really want to diagnose them with cancer, or vice versa. You wouldn't really expect a competent doctor to make such mistakes, but rather to make mistakes when there's more ambiguity and uncertainty, where multiple conditions may meet the symptoms (which then arguably wouldn't be a mistake), or they'd miss something, or it's some rare disease. The mistakes of ChatGPT might be more arbitrary.
Understanding how a method works and why and how individual outputs are generated could also increase trust.
As far as ChatGPT is concerned, we understand the low-level maths, and we understand what it does on a high level, but we can't explain why a individual outputs are generated (beyond "sticking this input into these equations give this output").
Unless something significant changes here, we'd probably need to rely primarily on reliable results, rather than understanding.
If anything, the understanding we have might decrease our trust, as we know that it just chains words together, based on what it's seen in the past, without much concern for nor ability to evaluate whether the result makes sense.
Side note: explainability is a big topic in AI, although the more complex models tend to fail there.
Also, the medical domain has a particularly high threshold for trust, as wrong decisions could very directly lead to people's deaths, and it doesn't have the benefit of, say, engineering, where every individual design can be reviewed and extensively tested before anyone's lives depend on it working (never mind failsafes).
Other domains may very well meet the threshold for trust before the medical domain would.
I could also see something like ChatGPT being used or supervised by a doctor, to aid diagnosis more than to diagnose by itself. It might also be used (mostly by private individuals, probably) in cases where doctors aren't available (although something manually curated, like WebMD, is probably better for that, even though that is not without problems).
