The Responsible Organisation

The Centre de Telecomunicacions i Tecnologies de la Informació (CTTI) is the central IT provider for the Government of Catalonia, responsible for managing all digital services. It is a public company with 500 employees that manages all digital services for the government, including telecommunications, workplace technology, applications, and data centers. 

In the case of the generative AI project for summarising legal texts, CTTI played a coordinating role, bringing together the different partners and providing technical expertise. They also ensured that the project aligned with the government's IT strategy and innovation goals.

The Publications Office of the Government of Catalonia is responsible for publishing all legal texts in Catalonia. With regards to the Gen AI tool to summarise legal texts, the Publications Office is the responsible organisation where the solution was implemented, having initiated the project by seeking a solution to make legal texts more accessible to citizens in collaboration with CTTI.

Accenture, the global consulting and technology company, was the Publications Office’s existing web development partner and, in such condition, they were contracted to implement the Generative AI solution as an extension of their current contract.

Microsoft, a global technology provider, supplied the underlying AI technology through their Azure OpenAI Service. The solution utilized OpenAI's large language models (LLMs), initially GPT-3 and later GPT-4, to generate the summaries.

This article is based on insights gathered from an interview with Daniel Marco i Pàrraga, Director of Innovation, and Joseph Ignasi Bonet Pocino, Digital Innovation Dinamization Manager from the CTTI.

The problem

The tool addresses the primary problem of low understanding and accessibility of legal texts by citizens due to their complex and technical language. Legal complexity creates a barrier between the law and citizens, hindering their ability to understand, engage and interact with the legal system effectively. 

As explained by Mr. Marco and Mr. Bonet, citizens often struggle to decipher the legal terminology, and the sentence structures commonly found in legal documents, therefore having a low understanding of laws and regulations. This, in turn, reduces awareness of their rights and obligations, limiting their participation in the legal system and discouraging them from actively engaging with legal resources or processes. The perception of the law as inaccessible and incomprehensible can lead to a lack of trust in the legal system and a sense of detachment from the law-making process. 

The solution and its implementation

As a response to the above-described communication gap between the legal system and the citizens it serves, the Government of Catalonia implemented a generative AI tool to summarise legal texts in a plain and simple language that is simpler to understand for non-technical users. This was achieved through a collaborative effort involving the Centre de Telecomunicacions i Tecnologies de la Informació (CTTI), the Publications Office of the Government of Catalonia, Accenture, and Microsoft. 

The solution was developed by using OpenAI's large language models (LLMs), initially GPT-3 and later GPT-4, and customising them with a unique prompt to ensure consistent and accurate summaries of the laws. The development process was divided into two main phases, consisting of the proof of concept and the scale-up.

Proof of Concept (PoC)

The Publications Office was inspired by the release of ChatGPT (November 2022) and wanted to explore the use of AI to summarise legal publications in plain language. Subsequently, a team was formed with members from CTTI, the Publications Office, and Accenture to test the feasibility of using generative AI tools for this purpose. 

The PoC lasted for three months, during which three main steps were carried out: testing the technology, prompt engineering, and conducting quality control. First, the team tested the technology and assessed the feasibility of using Generative AI to summarise legal texts accurately and consistently. Second, a major effort was dedicated to developing a unique prompt or instruction that could guide the AI model to accurately summarise any legal publication without hallucinations, i.e. without generating inaccurate or misleading information that is not supported by the input data or real-world knowledge. To test this, the team tested the prompt on 44 laws that were strategically selected, as these documents are publicly available, thus eliminating concerns about data privacy.  Finally, a quality and performance review process were conducted to ensure the accuracy and quality of the generated summaries. The team considered the "professional user" (lawyers) as the initial target audience for the tool, believing they would be better testers for this first phase. For this reason, lawyers from each participating organization reviewed the summaries of the 44 selected laws, providing feedback and identifying areas for improvement. The PoC concluded that the technology was promising and could be used to develop a production-ready solution. 

Project and Scale-up

After successful testing, a formal project was initiated to put the solution into production. Accenture, the Publications Office existing web development partner, was contracted to implement the solution as an extension of their current contract. 

Initially, GPT-3 was used, but the team later switched to GPT-4 due to its superior performance. The Catalan language was included in the base LLM models. However, the team found that prompting in English and requesting responses in Catalan or Spanish yielded better results. Moreover, lawyers from Accenture, the Publications Office, and CTTI reviewed the summaries of the initial 44 laws to ensure accuracy and identify any inconsistencies. Finally, the project was scaled up to include all 14,000 laws under the Publications Office’s responsibility.

Throughout the development process, the team prioritised data privacy and transparency. Legal texts were chosen as they are publicly available, eliminating privacy risks. A disclaimer was added to the website to inform users that the summaries are AI-generated for informational purposes only and do not hold legal value.

Expected benefits

Catalonia’s generative AI tool for summarising legal texts offers several benefits, among which the following:

• Enhanced users’ comprehension: By summarising legal texts in simple, non-technical language, the solution makes them more accessible and understandable for the users. This can empower citizens to better understand the legislation and their rights and obligations, potentially contributing to increased engagement with the legal system.
• Automation and increased efficiency: The solution automates the summarisation process of a large number of legislative publications, a task that would not be possible to be conducted by humans, according to the Mr. Marco and Mr. Bonet.
• Data privacy: The development of this tool does not pose any risks on data privacy, as the model uses exclusively already published and publicly available legal texts, without using personal data.
• Transparent and trustworthy innovation in public services: The solution demonstrates the potential of generative AI to improve public services, as Mr. Marco explains: "It's a very relevant project because we tested generative AI in an environment and it's not risky for anyone”. It serves as a successful example of how AI can be used to bridge the communication gap between the government and its citizens, employing AI tools in a controlled environment.  

Main challenges 

The main challenges encountered during the development and implementation of the generative AI solution for summarising legal texts were:

• Data consistency: A relevant data-related challenge was the quality and consistency of the legal texts used to train and evaluate the generative AI model. Although the large amount of published legal texts available was crucial to tailor and test the prompt, their inconsistencies were particularly challenging. Specifically, according to Mr. Marco, “the problem wasn't the quality of the model, the problem was the quality of the law, the 44 laws were written with a different quality, so it's very difficult to achieve a standard level of quality when you don't have a standard level of quality in the source”. In fact, besides their complex terminology, legal texts are often written by different authors with varying writing styles and levels of clarity and contrasting structures. This inconsistency can make it difficult for the AI model to learn a consistent pattern for summarisation. 

• Mitigating hallucinations: The risk of the AI model generating hallucinations, inaccurate or misleading information was mitigated by designing the prompt and conducting testing with legal experts. Developing a unique prompt was a significant challenge and, to ensure the accuracy of the tool, the project team focused on prompt engineering. Specifically, the team solved this challenge by conducting a Proof of Concept where they iterated the prompt drafting and tested it to achieve one unique prompt. 

• Promoting correct and responsible use: Another significant challenge was the need of ensuring that citizens and legal professionals understand the purpose and limitations of the AI-generated summaries, and that they use the tool responsibly. The incorrect use of the summaries, in effect, could potentially cause citizens’ misinterpretation of the summaries as official legal documents or substitute them for professional legal advice. Furthermore, legal professionals might over-rely on the summaries, potentially overlooking crucial details in the full legal text, which leads to the risk of the content being used inappropriately in court or other legal proceedings. Communicating the value of the solution and its correct use to promote its adoption by citizens and legal professionals was, therefore, crucial. For this purpose, disclaimer was included on the website explicitly stating that the summaries are for informational purposes only and do not hold legal value. In the interview, Mr. Marco acknowledges that while the disclaimer might seem unnecessary to some users, it is crucial for protecting the integrity of the legal system and preventing misunderstandings. 

Link: https://govern.cat/salapremsa/notes-premsa/622202/el-govern-utilitza-el-llenguatge-planer-per-facilitar-la-comprensio-dels-textos-juridics-a-la-ciutadania 

Contact: daniel.marco@gencat.cat