Have you heard of the powerful tool that is taking the world by storm? ChatGPT, a large language model developed by OpenAI, has been making waves lately for its ability to generate human-like responses to natural language prompts. Numerous recent articles on the internet report how ChatGPT passed the law bar exam in the US – although various commentators and journalists do acknowledge that this is very different from actually practicing and applying the law. But just how reliable is this technology? We put ChatGPT to the test by asking it some legal questions and were impressed by its capabilities.
However, it is important to keep in mind that while ChatGPT is impressive, it is not reliable, even with regard to straightforward legal questions, let alone nuanced factual and legal scenarios. As we discovered when testing it with some legal questions, there are major limitations to its abilities. A world in which technologies like ChatGPT replace lawyers and tried-and-tested legal research methodologies and databases, seems far away. As mentioned in its frequently asked questions, ChatGPT 3.5 is not connected to the internet and its knowledge is limited to events before 2021. It can also occasionally produce incorrect or biased responses, which could have dire consequences.