Reduced source to standard mapping preparation effort
The prototype reduced the preparation required before experts reviewed source to standard mappings.
Confidential AI case study
Clinical Data Harmonization
Hybrid RAG and fine tuned small language models for explainable source to standard clinical data mapping.
A healthcare / clinical research organization needed to harmonize complex source datasets into a standardized data model for downstream analytics, reporting, and research workflows. The source data contained a large number of variables with inconsistent naming conventions, incomplete descriptions, mixed data types, missing value codes, and domain specific clinical meanings.
Project classification: This proof of concept was developed for a confidential healthcare and clinical research organisation.
Generated structured and reviewable harmonisation plans
It produced structured harmonisation plans that domain experts could inspect, correct and approve.
Improved candidate retrieval through hybrid RAG
Hybrid semantic, keyword and metadata retrieval improved candidate selection when variable names did not match directly.
01 / Business context
Who needed the solution
A healthcare / clinical research organization needed to harmonize complex source datasets into a standardized data model for downstream analytics, reporting, and research workflows. The source data contained a large number of variables with inconsistent naming conventions, incomplete descriptions, mixed data types, missing value codes, and domain specific clinical meanings. Manual harmonization required significant domain expertise and was time consuming, especially when variables required mapping, renaming, transformation logic, category conversion, or new derived variable creation. To protect confidentiality, client name, dataset name, project name, and standard model details are not disclosed.
02 / Challenge
What needed to change
The organization wanted to reduce the manual effort involved in mapping raw clinical variables to a target standard. The main challenges were: • Source variables were often ambiguous or poorly documented. • Similar clinical concepts appeared with different names across datasets. • Some variables required direct mapping, while others required transformation logic. • Traditional keyword matching was not enough because clinical meaning depends on context. • LLM outputs needed control because they could over predict mappings or generate unnecessary transformations. • The solution needed to be generalized, not hardcoded for one dataset.
Ambiguous or incomplete source metadataSource variables were often ambiguous or poorly documented.
Different names for the same clinical conceptSimilar clinical concepts appeared with different names across datasets.
Direct mappings and transformation logic in the same workflowSome variables required direct mapping, while others required transformation logic.
Keyword only matching missed clinical contextTraditional keyword matching was not enough because clinical meaning depends on context.
Uncontrolled model outputs could over predict operationsLLM outputs needed control because they could over predict mappings or generate unnecessary transformations.
03 / Workflow transformation
Before and after
Previous workflow
- Ambiguous or incomplete source metadata
- Different names for the same clinical concept
- Direct mappings and transformation logic in the same workflow
- Keyword only matching missed clinical context
- Uncontrolled model outputs could over predict operations
Structured workflow
- Enrich source metadata
- Retrieve hybrid candidates
- Generate mapping plans
- Validate and score outputs
- Review structured outputs
04 / Solution
How we approached it
We designed an AI assisted harmonization workflow using hybrid RAG, metadata enrichment, and fine tuned small language models. Instead of relying on a single large model, the solution used multiple specialized small LLM components fine tuned for different parts of the workflow.
Enrich source metadata
Profile names, descriptions, types and value patterns before retrieval.
Retrieve hybrid candidates
Combine semantic, keyword, metadata and domain aware matching.
Generate mapping plans
Use fine tuned small language models for mappings and transformations.
Validate and score outputs
Apply schema checks, guardrails, confidence filters and repeatable evaluation.
05 / Example workflow
From input to reviewable output
1Enrich source metadata
2Retrieve hybrid candidates
3Generate mapping plans
4Validate and score outputs
5Run validation and evaluation
6Domain expert review
06 / Delivery scope
What the delivery covered
Workflow and domain mapping
- Metadata profiling and enrichment
- Candidate target variable retrieval
- Structured harmonisation plans
- Expert review workflow
AI, retrieval and evaluation
- Hybrid semantic and BM25 retrieval
- Fine tuned small language models
- Schema validation and confidence filtering
- Batch evaluation and benchmarking
Data and engineering
- Python processing pipeline
- FAISS vector retrieval
- PyTorch and LoRA fine tuning
- Pandas and NumPy data handling
Scope boundary
The work was limited to a proof of concept focused on retrieval, mapping and validation. Production integration, the final target standard and externally approved benchmark reporting were outside this public scope.
07 / Architecture and controls
How the system is organised
Python
FAISS
BM25
Pydantic
PyTorch
Pandas
LoRA
NumPy
Schema and operation validation
Structured schemas, consistency checks and transformation rules reduce malformed or unnecessary operations.
Confidence aware review
Low confidence mappings remain visible for domain expert review rather than being silently accepted.
Source traceability
Candidate mappings retain the metadata and retrieved target context used to form the recommendation.
Confidentiality boundary
Client, dataset and target model details remain anonymised in the public case study.
08 / Business value
Value observed during validation
1Pilot observed
Reduced source to standard mapping preparation effort
The prototype reduced the preparation required before experts reviewed source to standard mappings.
2Pilot observed
Generated structured and reviewable harmonisation plans
It produced structured harmonisation plans that domain experts could inspect, correct and approve.
3Pilot observed
Improved candidate retrieval through hybrid RAG
Hybrid semantic, keyword and metadata retrieval improved candidate selection when variable names did not match directly.
4Pilot observed
Reduced unsupported operations through validation and guardrails
Schema checks and guardrails reduced unsupported mappings and unnecessary transformation steps.
5Pilot observed
Created a repeatable benchmark and batch evaluation workflow
The workflow created a repeatable benchmark and batch evaluation process for further model iteration.
09 / Technology
Every component, with its role in the delivery.
Technology choices from the supplied project brief, mapped to the workflow each component supports.
Python
AI, data processing and backend logic
FAISS
Fast vector similarity retrieval
BM25
Keyword and exact term retrieval
Pydantic
Structured schema and output validation
PyTorch
Model training and inference
Pandas
Tabular data processing and analysis
LoRA
Parameter efficient model fine tuning
NumPy
Numerical processing
Related work
Explore related case studies.
