Building a Governed AI Assistant for Citizen Casework with Azure Databricks

Building a Governed AI Assistant for Citizen Casework with Azure Databricks

Public-sector casework often begins with unstructured text. A resident describes a missed collection, a housing problem or a dangerous pothole in their own words. An officer then has to identify the service area, extract the important facts, judge whether the case may be urgent and find the relevant policy before deciding what happens next.

AI can help with that work, but only if the system is designed around evidence and accountability. A fluent answer is not enough. Officers need to know what came from the original case, which policy supports the advice and when the system did not find enough evidence to make a recommendation.

At Opsmorph, we built a Databricks AI Application proof of concept to explore that problem. The result is a governed, human-in-the-loop workflow on Azure Databricks that converts free-text citizen cases into reviewable evidence, retrieves relevant policy guidance and produces advisory recommendations without transferring decision authority to AI.

The POC also supports a second experience for service managers. The same governed data foundation feeds reconciled operational metrics and a Databricks Genie space, allowing managers to ask questions about backlogs, SLA pressure and resolution times in everyday language.

Everything in the demonstration is synthetic. The application uses fictional citizen cases and fictional policy guidance, makes advisory suggestions only and always leaves the final decision with a person.

What We Built

The application follows a staged evidence pipeline:

1. Ingest and validate raw citizen cases.
2. Classify each case into a service area.
3. Extract structured facts from the free-text description.
4. Retrieve relevant fragments from a governed policy index.
5. Recommend, escalate or abstain for a human case officer.
6. Publish reconciled metrics for management analytics and Genie.

Azure Databricks provides the common workspace for data, machine learning and AI workflows. Delta Lake stores reliable tables with controlled updates and history, while Unity Catalog provides permissions, governance and lineage across data and models. Databricks Workflows runs each stage in the correct order.

The AI and analytics layer combines several deliberately different tools. MLflow tracks and registers the classification model. Azure OpenAI extracts structured information and drafts grounded advice. Azure AI Search supports vector and hybrid policy retrieval. A serverless SQL warehouse and Databricks Genie provide the management experience.

Terraform and GitHub Actions make the deployment repeatable. GitHub authenticates to Azure through keyless OIDC, Terraform provisions the platform and source-controlled notebooks implement and validate each stage.

The important architectural choice is not any individual service. It is that both user experiences depend on one governed data foundation rather than maintaining separate copies of case data, AI outputs and management metrics.

Synthetic Data by Design

We wanted the POC to demonstrate the complete workflow without using real citizen information or implying that synthetic results represent council performance.

A deterministic PySpark notebook generates 7,500 fictional cases across seven service areas: Housing, Council Tax, Benefits, Waste, Highways, Planning and Other. Fixed complaint templates are combined with synthetic identifiers, dates, priorities, statuses and channels.

The dataset includes failure cases on purpose. Every 97th description is null, creating 77 deliberately invalid records. That gives the ingestion stage something concrete to reject and proves that the quality gates are doing real work rather than merely reporting success on clean input.

Policy content is synthetic too. Six fictional documents contain two sections each, producing 12 searchable policy chunks. Azure OpenAI creates 1,536-dimensional embeddings for those chunks, but it does not generate the policy text itself. No external PDFs or real policy documents are used.

This boundary matters. Synthetic data lets us test orchestration, governance and safety behaviour, but it does not establish performance on real cases. That requires representative data, formal approval and a separate evaluation programme.

Govern the Data Before Applying AI

The first stage is conventional data engineering because reliable AI output starts with reliable input.

Stage 1 establishes the quality boundary: 7,500 synthetic records enter the workflow, 77 invalid records are rejected and 7,423 curated cases are published.

Raw records are checked for required fields, normalised and deduplicated using the latest update before being published to governed Delta tables. Of the 7,500 generated cases, the 77 intentionally invalid records are rejected and 7,423 curated records continue through the pipeline.

That sequence creates a useful control boundary. AI services do not operate directly on an unchecked landing area, and downstream analytics reconcile to the same curated dataset used by the application.

The order of those checks matters. Required-field validation prevents incomplete cases from silently entering later stages. Normalisation makes values consistent enough to compare and aggregate. Deduplication keeps the latest known state of a case, rather than counting every update as a separate incident. Only after those deterministic checks have succeeded is the governed Delta output made available to classification, extraction and reporting.

This separation also makes failures easier to diagnose. A rejected source record is a data-quality issue; a low-confidence service label is a model issue; and a missing policy citation is a retrieval or evidence issue. Keeping those boundaries explicit avoids treating every problem as an AI problem.

The classifier then routes each case to one of the seven service areas. The POC uses a lightweight TF-IDF and logistic-regression model rather than a generative model. MLflow records the evaluation run and registered model version.

On the intentionally simple synthetic dataset, the model achieved accuracy and weighted F1 of 1.0 across 5,938 training and 1,485 test records. That is evidence that the pipeline works with the demonstration templates, not evidence of production accuracy. In the application, the label remains a routing suggestion and a person is responsible for the case.

Stage 2 uses TF-IDF and logistic regression for a narrow, explainable routing task, with the registered model and evaluation evidence tracked in MLflow.

Using a conventional classifier here is deliberate. The task has a small, fixed label set and does not require generated prose. A compact model is easier to version, evaluate and replace than a general-purpose language model, while still producing a confidence score that can be exposed to reviewers. In the illustrated case, the model suggests Highways with 99.3% confidence, but that score does not grant the system authority to route or close the case without human oversight.

For a production service, the evaluation would need to become substantially harder. Real case language includes abbreviations, spelling errors, mixed topics, incomplete descriptions and changes in service terminology. Testing would therefore need class-level precision and recall, confusion analysis, confidence calibration, drift monitoring and a clear threshold below which the system asks for review instead of presenting a strong routing suggestion.

Extract Facts Without Filling the Gaps

Classification answers one question: which service area is most likely responsible? Officers still need structured facts from the original description.

Azure OpenAI performs that extraction through a strict JSON schema. The response is validated before it is accepted, and the extraction rules require each value to be supported by the source text.

For example, the case “There is a dangerous pothole on the road near my home” supports a Highways classification and a health-risk flag. It does not explicitly state an urgency level. The correct structured value is therefore unknown, not an inferred severity that happens to sound plausible.

This is a small but important design principle: uncertainty must survive the transformation from free text to structured data. A system that silently replaces missing evidence with confident guesses makes later review harder, not easier.

The schema turns a probabilistic model response into a controlled application contract. Expected fields such as service, health risk, urgency, confidence, validation status and human-review status must be returned in the correct shape and type. A malformed response cannot quietly flow into policy search or management reporting as though it were valid data.

Schema validity and operational safety are not the same thing. A response may be structurally valid and still require a person to inspect it because evidence is incomplete or the scenario is sensitive. The validation summary in the POC therefore distinguishes errors, valid items that can continue and valid items that still require review. This is more useful than a single pass/fail flag because it preserves the reason a case was stopped.

The managed-identity connection to Azure OpenAI is part of the same control model. Application code does not need to carry long-lived service credentials, and access can be governed through the Azure and Databricks identity boundary. In production, that identity design would be complemented by private networking, approved model deployments, prompt and response logging rules, retention controls and explicit handling for personal data.

Ground Advice in Policy — or Abstain

The policy stage uses retrieval-augmented generation to find relevant chunks before advice is drafted. A supported question can return a grounded answer with citations to the accepted policy evidence.

An unsupported question follows a different path. If the retriever does not return acceptable evidence, the system responds with “Insufficient policy evidence.” It provides no recommendation and no citation.

That abstention behaviour is a product feature, not an error condition. For casework, an explicit gap is safer and more useful than an invented policy answer. It tells the officer that the automated path has reached its limit and that further investigation is required.

The implementation also keeps retrieval separate from recommendation. Vector search provides evidence; the agent decides whether the evidence is sufficient and how to summarise it. This makes the boundary observable and allows the validation notebooks to test citations, recommendations and abstentions independently.

A Bounded Agent, Not an Autonomous Decision-Maker

The case-officer agent gathers evidence through five bounded tools in a fixed sequence:

1. get_case reads the source case.
2. get_case_classification reads the model output.
3. get_case_extractions reads validated structured facts.
4. search_policy retrieves relevant policy evidence.
5. summarise_and_recommend produces a grounded recommendation or abstains.

The agent cannot update the source case and cannot make the final decision. Its job is to assemble evidence for review.

In the running example, a dangerous pothole is classified as Highways, the extraction identifies a health risk and policy search provides supporting evidence. The result is an urgent-escalation recommendation with a policy citation. A human officer still decides what action to take.

Three scenarios test the safety boundary: an ordinary supported recommendation, an urgent supported escalation and an unsupported case that must abstain. All 19 validation checks passed in the POC, including the requirement that the unsupported case returns neither a recommendation nor a citation.

One Data Foundation, a Second User Experience

Case officers work one case at a time. Service managers need a wider view: open workload, SLA breaches, resolution time and trends by service or region.

The POC publishes a documented analytics table from the curated data. Its 7,423 rows reconcile with the accepted case records, and 13 validation checks confirm the management dataset and metric definitions.

Databricks Genie provides a natural-language interface over that governed table. A manager can ask, “Which services have the most SLA breaches?” and receive an answer, a result table, a visualisation and inspectable SQL.

In the synthetic demonstration, Waste has the most SLA breaches at 1,061. The value is useful for validating the application experience, but it is not a statement about the performance of any real council.

Inspectable SQL is valuable here. Natural-language analytics should not create a second, opaque definition of a business metric. Managers and data teams need to see how the answer was produced and reconcile it with the governed source.

Validation Is Part of Every Stage

The repository is the source of truth for infrastructure, notebooks, configuration and documentation. Validation lives beside each stage as a dedicated notebook rather than in a separate test folder.

The developer view keeps infrastructure, application stages, validation notebooks and architecture decisions under version control.

This keeps delivery evidence close to the code it checks. The CI/CD path deploys infrastructure and notebooks, triggers the Databricks jobs and records stage-specific PASS or FAIL results.

The repository layout makes ownership visible. GitHub workflow definitions describe the delivery path. Terraform in infra/ defines the Azure and Databricks resources. Application notebooks and Genie configuration live under src/, while docs/ and architecture decision records capture the reasoning behind important choices. Validation notebooks sit alongside stages 1–6 so that a change to extraction, retrieval or analytics can be reviewed together with the evidence that proves it still behaves as intended.

Deploying from Git also reduces configuration drift. A notebook edited only in the online workspace can be difficult to reproduce and review; a version-controlled notebook can pass through the same pull-request, deployment and validation process as the infrastructure around it. Keyless OIDC authentication between GitHub and Azure further avoids making a long-lived cloud secret part of that delivery chain.

The tests go beyond confirming that resources exist. They check whether invalid records are rejected, analytics rows reconcile, extracted fields conform to their schema, citations are present only when evidence supports them and unsupported scenarios abstain safely.

This is especially important for AI systems. Infrastructure tests can prove that a model endpoint and vector index are available. They cannot prove that the application respects an evidence boundary when the right answer is “I do not know.”

What the POC Demonstrates

The POC demonstrates a governed end-to-end architecture with keyless AI access, policy-grounded citations, safe abstention, human review, audit lineage and reconciled management analytics.

It also demonstrates a practical division of responsibility:

• Deterministic data engineering validates and curates the source data.
• A lightweight classifier handles a narrow routing task.
• Generative AI extracts facts and drafts advice within explicit schemas and evidence rules.
• Retrieval supplies policy context and citations.
• The agent coordinates bounded tools but cannot modify cases or make decisions.
• Humans remain accountable for the outcome.

Together, these capabilities establish a credible foundation for a controlled pilot. The architecture, automated deployment path, evidence rules and validation approach exist as working components rather than only as a conceptual design. They can be extended without changing the central control model: governed inputs, traceable outputs and a person accountable for every decision.

The most valuable next step is to apply that foundation to an approved, representative dataset and integrate it with a real casework workflow. That would allow case officers and service managers to evaluate usefulness in context while the same human-review and evidence controls remain in place.

Final Thoughts

The most useful outcome of this POC is not a chatbot. It is an evidence pipeline that makes every automated step reviewable.

Citizen text becomes governed data. Model outputs remain suggestions. Extracted facts preserve uncertainty. Policy-backed advice includes citations, unsupported questions trigger abstention and every recommendation stops with a human decision-maker. The same curated foundation then supports management analytics without creating a separate version of the truth.

That is a stronger starting point for public-sector AI than optimising for autonomy. The next sensible step is a controlled production-discovery phase focused on real-data approval, representative evaluation, workflow integration and operational ownership — with human accountability remaining the design anchor.