ÌìÃÀÓ°Ôº

Tier 1 Information

Name

Money Information Centre Chatbot

Description

The purpose of the AI-powered chatbot is to provide a better search experience for visitors of the Money Information Centre (‘MIC’) website. The chatbot is built on AWS Bedrock and provides answers to visitors queries about information on the website. It uses data that is publicly available on the MIC website to provide responses.

The chatbot is introduced to improve accessibility to up-to-date financial guidance on the MIC website and to enhance the user experience of the website.

Website URL

Contact email

MICfeedback@leeds.gov.uk

Tier 2 - Owner and Responsibility

1.1 - Organisation or department

Leeds City Council

1.2 - Team

Financial Inclusion Team

1.3 - Senior responsible owner

Chief Officer Community Hubs, Welfare and Business Support

1.4 - External supplier involvement

Yes

1.4.1 - External supplier

Kainos Software Ltd

1.4.2 - Companies House Number

Company number NI019370

1.4.3 - External supplier role

The supplier Kainos Software Ltd (‘Kainos’) developed the Money Information Centre Chatbot on AWS Bedrock. The supplier supported the restructuring of data, developed the system prompts to the chatbot, provided initial testing and infrastructure of temporary monitoring of performance during the pilot period.

1.4.4 - Procurement procedure type

Fully funded partner investment through agreement with Amazon Web ÌìÃÀÓ°Ôº.

1.4.5 - Data access terms

Kainos Software plc provides customer services to Leeds City Council and has full data access in order to fulfil the service requirements. The solution processes data from Money Information Centre (‘MIC’) website that is available in the public domain. Chatbot conversation transcripts will be kept during the trial period. There will only be a window of time where this data might contain (unsolicited) personal data prior to a redaction process. Kainos Software Ltd will not have access to the unredacted scripts.

Tier 2 - Description and Rationale

2.1 - Detailed description

The MIC chatbot leverages a Retrieval Augmented Generation (RAG) architecture to answer visitor questions using website content. All website data was ingested by extracting text and key metadata, then segmented into manageable chunks. Each chunk is transformed into a high-dimensional vector embedding with an Amazon titan text-embedding model, ensuring that semantically similar content produces similar vectors. These embeddings are stored in a vector database (Amazon OpenSearch) optimised for fast approximate nearest neighbour search.

When a visitor submits a question, the system converts the query into an embedding and performs a cosine similarity search against the vector database to retrieve the most pertinent chunks. These chunks, along with their metadata, are aggregated and formatted into a comprehensive prompt, which is then passed to a large language model. The model generates a precise, contextually grounded response by combining its internal training with the externally retrieved, domain-specific data in tandem with defined Amazon Bedrock guardrail checks, thereby reducing hallucinations and enhancing accuracy.

2.2 - Scope

The chatbot has been designed to help visitors find relevant information listed on the MIC website. This includes contact information for agencies and organisations that help in relation to money, debt and benefits, housing, energy, fuel and food and employment matters.

The chatbot is explicitly not designed to give financial advice and/or to provide any information that is not already available on the MIC website. It is also not designed at present to escalate to a human agent. Guardrails have been introduced to keep the chatbot on topic and to warn against the potential misconception of there being a human on the other side of the conversation.

2.3 - Benefit

The purpose of the chatbot is to: 1) Improve accessibility to trusted, up-to-date financial guidance on Leeds City Council’s Money Information Centre (MIC) website 2) Enhance the user experience of the MIC website 3) Explore a solution that might be scalable across Leeds City Council websites, beyond only the MIC website

A feedback form is introduced and an evaluation plan is in place to assess at the end of a pilot period whether benefits are being achieved and the chatbot is performing as intended.

2.4 - Previous process

Prior to piloting the tool, visitors had the option to navigate the website, to use the search functionality on the website and/or to contact an agent to ask questions. None of these functionalities have been removed, the chatbot is an added functionality.

2.5 - Alternatives considered

A solution was considered that would be pure semantic search, however, this was not believed to have the same potential benefit of an enhanced user experience as the selected solution. In addition, ‘doing nothing’ was considered. This approach was chosen because it has the best chance of achieving the aspired benefits of improved search and enhanced user experience.

Tier 2 - Decision making Process

3.1 - Process integration

The MIC chatbot will be integrated into the MIC website and it is limited to providing visitors with information available on the MIC website. Engagement with the chatbot will be entirely voluntary and at the onset, the Leeds City Council contact number is provided as an alternative route of contact. No formal decisions will be made based on interaction with the MIC chatbot. The chatbot user can decide to act upon the information provided or not.

3.2 - Provided information

The tool provides responses in text format calling upon MIC webpages and their corresponding guidance to the user for review. This is shared with the user as a text response and contains a response in a natural language way with links to the content. No images, videos or graphs are provided only text based responses and URL links to content. Upon starting their engagement with the chatbot, users are informed to always check content themselves given the risk that information that the chatbot provides could be inaccurate.

3.3 - Frequency and scale of usage

Usage numbers of the MIC chatbot will need to be observed over time. The MIC website currently has around 50,000 hits a year.

3.4 - Human decisions and review

When the tool provides the responses to the user, it is up to the user to decide if the response has provided them with satisfactory information in relation to the question they were asking. They can ask follow up questions to obtain further information that may help them answer their questions or ask new questions.

It is left to the human to decide if their question has been answered to satisfaction and the tool makes no decisions on completion of the user query. There is a plan in place to review the quality of user support at the end of a pilot period, in addition to extensive testing that was conducted before bringing the chatbot live.

3.5 - Required training

The development team has worked closely with the Leeds City Council Innovation team and the Financial Inclusion team to understand, deploy and configure the MIC chatbot.

3.6 - Appeals and review

If the chatbot user is not satisfied with the information provided by the chatbot they have the option to contact Leeds City Council directly.

A feedback form is also provided. Once completed, feedback will be e-mailed to a designated mailbox using Amazon Simple Notification Service, ensuring secure and reliable email delivery.

Tier 2 - Tool Specification

4.1.1 - System architecture

The MIC chatbot leverages a serverless architecture that integrates Amazon Bedrock with a knowledge base built on Amazon OpenSearch Service, enabling robust Retrieval Augmented Generation (RAG) capabilities. In this setup, the chatbot retrieves and processes the MIC website’s ingested content to generate intelligent, accurate, and context-aware responses. The solution is designed for immediate needs while offering scalability and flexibility to incorporate additional LCC digital services, documents, and datasets in the future.

The chatbot is hosted as a stand-alone application on Amazon Amplify and can be embedded into the MIC Drupal website using either an iframe or a custom embeddable script served from Amazon S3 (and distributed via CloudFront) and calling Amazon Lambdas. User feedback is captured securely through Amazon Simple Notification Service (SNS), ensuring continuous improvement of the tool.

4.1.2 - Phase

Beta/Pilot

4.1.3 - Maintenance

Functionality is implemented to allow for the chatbot to automatically incorporate any updates on the MIC website on a [24-hour] schedule, using AWS Event Bridge and AWS Lambda. Any new content added to the website by the provider would therefore be automatically ingested into the chatbot’s knowledge base.

In addition to our automated update process, many components of the solution are built on AWS managed services, which handles the responsibility for software concurrency and maintenance. At the same time, we maintain strict software currency controls through Continuous Integration/Continuous Deployment pipelines, ensuring that all packages used by the chatbot are automatically audited for vulnerabilities.

In the first instance, the MIC chatbot is only implemented for a 6-week pilot period to collect feedback and data around its performance. During the pilot, the operator will review user feedback and conversation transcripts to implement any necessary adjustments or enhancements for optimal performance and relevancy. Review at the end of this pilot period will inform stable and/or wider roll-out and an associated maintenance schedule.

4.1.4 - Models

The tool incorporates two primary machine learning models:

Claude 3 Sonnet: This is the large language model (LLM) responsible for generating intelligent, context-aware responses in the chatbot.

Titan Text Embeddings V2: This model converts website content and user queries into high-dimensional vector embeddings, enabling efficient retrieval through our RAG lookup process.

Tier 2 - Model Specification

4.2.1 - Model name

Claude (Chatbot), Titan Text Embeddings (RAG)

4.2.2 - Model version

Claude 3 Sonnet (Chatbot), Titan Text Embeddings V2 (RAG)

4.2.3 - Model task

To use natural language understanding to interpret and process information the customer provides as input and provide a output answer

4.2.4 - Model input

Text based questions in English

4.2.5 - Model output

Conversational response to the user defined question in text format

4.2.6 - Model architecture

Claude 3 Sonnet: A transformer-based LLM optimised for natural dialogue, using multi-head self-attention and deep feed-forward networks. It employs reinforcement learning from human feedback to improve response quality.

Titan Text Embeddings V2: A transformer-based model that converts text into high-dimensional vectors for similarity search in RAG, ensuring fast, accurate data retrieval by encoding semantic meaning into dense vectors.

4.2.7 - Model performance

Testing was conducted with input from domain experts. On average, 83% of chatbot responses were scored at least a 3 out of 5 in quality.

Fairness testing did not reveal any stable biased patterns.

4.2.8 - Datasets

MIC website content data: public data on the MIC website

For testing, a set of frequently asked questions combined with human generated answers (verified by domain experts) and chatbot answers were generated.

4.2.9 - Dataset purposes

The frequently asked questions dataset was used for testing.

No AI model was trained to create the MIC chatbot. The chatbot leverages a pre-trained model.

Tier 2 - Data Specification

4.3.1 - Source data name

Chatbot knowledge base

4.3.2 - Data modality

Text

4.3.3 - Data description

Information on the MIC website which is in the public domain. It contains details of free advice and support services available in Leeds. These services can assist with a range of money-related matters, such as: - Debt - Benefits - Housing - Energy and utilities

4.3.4 - Data quantities

The team ingested all publicly accessible text from the Money Information Centre website into the Bedrock knowledge base via the default parser. As this approach doesn’t involve traditional training, validation, and test splits, the entire website content —approximately a few megabytes of HTML text— was processed for retrieval-augmented lookups rather than model development in a conventional train/test manner.

4.3.5 - Sensitive attributes

None

4.3.6 - Data completeness and representative-ness

The data within the knowledge base comprises the complete dataset on the MIC website.

4.3.7 - Source data URL

and all its subpages. The MIC chatbot only have access to content of the MIC webpages and does not have access to the content stored on the websites of agencies that the MIC website lists.

4.3.8 - Data collection

All information used is held on the MIC website - information is not used from any other source.

4.3.9 - Data cleaning

N/A

4.3.10 - Data sharing agreements

All the information being used by the AI Chatbot is the Council’s existing website content that is publicly available over the internet today. Any logging or observability data being retained by the service will remain in the Leeds City Council environment for service improvement analysis and will not be shared with third parties.

4.3.11 - Data access and storage

Leeds City Council will keep records of chatbot conversations for 30 days. This information will be used to inform the evaluation of the pilot period.

Users are discouraged from entering personal or private information into the chatbot by an opening message. Any personal data that is nevertheless volunteered will be redacted by Leeds City Council before further analysis.

Tier 2 - Risks, Mitigations and Impact Assessments

5.1 - Impact assessment

Data protection impact assessment - completed Feb 2025 Algorithmic impact assessment - completed March 205

5.2 - Risks and mitigations

The main risks identified include: - The risk of the chatbot providing misinformation. Mitigations include the implementation of AWS guardrails and the notification to chatbot users that they should verify content with links provided. - Vulnerable populations being misdirected or not receiving critical advice. The chatbot was thoroughly tested on this component and will refer to emergency helplines when detecting distress. A warning is provided at the opening of the chatbot window not to use the chatbot to report an incident or in an emergency, with a recommendation to call 999 instead. - A malicious actor corrupting the chatbot. A thorough security review was conducted and potential vulnerabilities were addressed. - Chatbot users mistakenly believing there is a human on the other side of the conversation. A message was introduced at the opening window saying You’re chatting to a computer. For further help, call Leeds City Council on 0113 222 444.