1. Client Context
A healthcare / clinical research organization needed to harmonize complex source datasets into a
standardized data model for downstream analytics, reporting, and research workflows.
The source data contained a large number of variables with inconsistent naming conventions,
incomplete descriptions, mixed data types, missing value codes, and domain-specific clinical
meanings. Manual harmonization required significant domain expertise and was time-consuming,
especially when variables required mapping, renaming, transformation logic, category conversion,
or new derived variable creation.
To protect confidentiality, client name, dataset name, project name, and standard model details
are not disclosed.
2. Problem
The organization wanted to reduce the manual effort involved in mapping raw clinical variables
to a target standard.
The main challenges were:
- Source variables were often ambiguous or poorly documented.
- Similar clinical concepts appeared with different names across datasets.
- Some variables required direct mapping, while others required transformation logic.
- Traditional keyword matching was not enough because clinical meaning depends on context.
- LLM outputs needed control because they could over-predict mappings or generate unnecessary transformations.
- The solution needed to be generalized, not hardcoded for one dataset.
The key goal of the POC was to build an AI-assisted harmonization pipeline that could produce
accurate, explainable mapping recommendations and reach the required benchmark score.
3. AI Approach
We designed an AI-assisted harmonization workflow using hybrid RAG, metadata enrichment,
and fine-tuned small language models.
Instead of relying on a single large model, the solution used multiple specialized small LLM
components fine-tuned for different parts of the workflow. These models were integrated with
retrieval, validation, and rule-based quality controls to produce structured harmonization plans.
The approach included:
Metadata Enrichment
Raw source metadata was enriched before retrieval and generation. The enrichment process used
available variable information.
This helped the system create stronger context for each variable before searching for possible
target matches.
Hybrid RAG Retrieval
A hybrid retrieval approach was used to improve candidate selection.
It combined:
- Semantic vector search
- Keyword/BM25-style search
- Metadata-based matching
- Domain-aware matching
- Data type and value-pattern comparison
This allowed the system to retrieve more relevant target-standard candidates, even when the
source and target variable names were not directly similar.
Fine-Tuned Small LLMs
Fine-tuned small LLMs were used to generate mapping and transformation recommendations.
The models were trained and guided to understand harmonization.
Validation and Guardrails
To reduce hallucination and over-prediction, additional validation logic was added after model
generation.
This included:
- Operation-level checks
- Schema validation
- Confidence-based filtering
- Mapping consistency checks
- Transformation sanity checks
- Prevention of unnecessary operations
- Structured JSON output validation
The final output was a structured execution plan that could be reviewed by domain experts.
4. Tech Used
The POC used a combination of AI, retrieval, data engineering, and evaluation components.
Core AI Technologies
- Fine-tuned small language models for clinical data harmonization tasks
- Open-source LLMs adapted for source-to-standard mapping and transformation-plan generation
- LoRA / parameter-efficient fine-tuning to specialize models without requiring full model retraining
- Retrieval-Augmented Generation to ground model outputs in standard metadata
- Hybrid RAG combining semantic search, keyword search, and metadata-aware retrieval
- Embedding-based semantic search for matching source variables to target-standard concepts
- LLM-based transformation planning for mapping, renaming, category conversion, null handling, data type conversion, and derived variables
- Guardrail-based validation to reduce hallucination and unnecessary operation prediction
- Structured JSON generation for machine-readable execution plans
Model and Retrieval Stack
- Small open-source LLMs fine-tuned for harmonization-specific tasks
- Embedding models for semantic similarity search
- Vector search using FAISS for fast candidate retrieval
- BM25 / keyword retrieval for exact-term and abbreviation matching
- Hybrid ranking logic combining semantic similarity, keyword overlap, domain relevance, data type compatibility, and value-pattern similarity
- Candidate filtering and reranking to improve mapping precision
- Multi-step AI workflow where retrieval, generation, validation, and scoring were handled as separate controlled stages
Programming and Data Engineering Stack
- Python as the primary development language
- Pandas for tabular data processing, profiling, and evaluation reports
- NumPy for numerical operations and metrics handling
- JSON / CSV / Excel processing for metadata, model outputs, ground truth, and reports
- Pydantic for schema validation and structured output parsing
- Regular expressions and custom parsers for cleaning, normalization, and transformation extraction
- Batch processing pipelines for running hundreds of variables through the harmonization workflow
- Logging and debugging framework to track retrieval results, model outputs, parsing failures, and evaluation behavior
Machine Learning and LLM Engineering Stack
- PyTorch-based model execution
- Hugging Face Transformers for working with open-source language models
- PEFT / LoRA-style fine-tuning for efficient model adaptation
- vLLM-style inference optimization for faster local model serving
- GPU-based inference for scalable batch execution
- Prompt engineering and task-specific instruction tuning
- Model output post-processing to enforce valid operation types and structured execution plans
Evaluation and Quality Measurement
- Ground-truth based evaluation against manually validated mappings
- Precision, recall, and F1 scoring at operation level
- Subset-based evaluation for mapped variables, new variables, and full datasets
- Error analysis across experiment cycles to identify over-prediction, under-prediction, and retrieval failures
- Operation-level scoring for mapping, rename, prefix handling, category conversion, formula logic, null conversion, and data type conversion
- Automated report generation using CSV and Excel outputs
- Experiment comparison framework to measure improvement across pipeline versions
5. Outcome / Business Value
The POC successfully reached the required target benchmark, demonstrating that AI-assisted
clinical data harmonization is feasible and valuable.
The solution provided several business benefits:
- Reduced manual effort in source-to-standard variable mapping.
- Generated structured, reviewable harmonization plans.
- Improved retrieval quality using enriched metadata and hybrid RAG.
- Reduced model hallucination through validation and guardrails.
- Created a repeatable evaluation framework for measuring improvement.
- Supported batch processing across large sets of variables.
- Helped domain experts focus on review and validation instead of starting mappings from scratch.
The POC proved that a well-designed combination of fine-tuned small LLMs, hybrid RAG,
guardrails, and automated evaluation can support scalable clinical data standardization.
6. What Similar Companies Can Learn
Companies working with healthcare, clinical research, life sciences, or regulated data can learn
several important lessons from this POC.
First, raw LLM output alone is not enough for reliable data harmonization. The best results come
from combining LLMs with strong retrieval, enriched metadata, validation rules, and measurable
evaluation.
Second, smaller fine-tuned models can be effective when they are trained for specific tasks and
integrated into a controlled workflow. This can reduce cost, improve control, and make the system
easier to deploy in enterprise environments.
Third, hybrid RAG is critical for data harmonization. Semantic search helps find conceptually
similar variables, while keyword and metadata search help preserve exact clinical and technical
signals.
Fourth, explainability matters. The system should not only suggest a target mapping but also provide
structured reasoning, transformation steps, and confidence signals so that human reviewers can
validate the output.
Finally, companies should treat AI harmonization as an AI-assisted workflow, not a fully autonomous
replacement for domain experts. The strongest value comes from accelerating expert review,
reducing repetitive manual work, and improving consistency across datasets.
