AI/ML 
Rozwój oprogramowania 
How we built an AI system for tongue analysis based on TCM
Client
The client develops digital diagnostic services in the field of Traditional Chinese Medicine (TCM), where tongue analysis is one of the key elements in assessing a patient’s health. In practice, this involves working with images and interpreting subtle visual features such as color, coating, cracks, or teeth marks. These features are then mapped to a set of diagnostic syndromes.
Until now, the entire process was performed manually by an expert. This limited scalability because the number of analyses depended directly on the availability of specialists. It also extended turnaround time and introduced natural variability in interpretation. The client needed a solution that would increase process throughput without compromising quality, and ultimately enable partial or full automation.
Challenge
The biggest challenge was not the use of deep learning itself, but building a meaningful AI pipeline in conditions far from “laboratory-like.”
The input data was unstructured. Tongue images and their descriptions existed separately, often scattered across email threads, and the descriptions themselves were written in free-text form. This meant there was no ready-made training dataset and it had to be built from scratch, including label extraction using NLP and expert validation.
Additionally, the dataset was limited in size and highly imbalanced. Some syndromes were rare, which made it difficult to train models using a standard classification approach. The problem was further complicated by the fact that the ground truth was not fully objective. Interpretation depended on the expert, and some symptoms were inherently difficult to label consistently.
From a visual perspective, the challenge was also significant. Images varied in quality, lighting, and angle, while diagnostic features were often subtle and low-contrast. In practice, this meant that a standard end-to-end classification approach quickly reached its limits.
Solution
Instead of a single classification model, we designed a pipeline aligned with expert reasoning.
1. Data structuring (NLP + human-in-the-loop)
- extraction of symptoms from textual descriptions,
- creation of a feature dictionary and mapping symptom to syndrome,
- iterative expert validation.
2. Model architecture (CNN + transfer learning)
We compared two approaches:
A. End-to-end syndrome classification
Fast, but with low interpretability and weaker performance.
B. Three-stage pipeline (final approach):
- symptom detection (multi-label),
- tongue region analysis (feature localization),
- syndrome classification.
Why this worked better:
- breaking the problem into simpler sub-tasks,
- better generalization with a small dataset,
- ability to validate intermediate outputs.
3. Optimization
- transfer learning using pretrained CNNs,
- class balancing,
- image data augmentation,
- iterative expert validation with a feedback loop.
4. Deployment (production-ready PoC)
- AWS based scalable inference pipeline,
- event-driven processing triggered by image uploads,
- result visualization with highlighted symptoms on images,
- human-in-the-loop mode with expert validation.
Results
At the Proof of Concept stage, the system achieved 80 to 90 percent accuracy in predicting TCM syndromes, while maintaining high precision in detecting individual symptoms.
Importantly, the model remains stable despite a limited and imperfect dataset, which confirms the validity of the chosen architecture.
From an operational perspective, the solution was successfully integrated into the expert workflow and reduced the time required to analyze a single case. This enabled handling a larger number of patients without a proportional increase in resources.
The system architecture is designed for further development, both in terms of scaling data and iterative model improvement. With dataset growth and continued optimization, the potential to reach accuracy levels close to 99 percent has been identified.
Ultimately, the system can evolve into a fully automated mode, where the role of the expert is reduced or eliminated, and diagnostic results are delivered directly to the patient.
Design, Development, DevOps czy Cloud – jakiego zespołu potrzebujesz, aby przyspieszyć pracę nad swoimi projektami? Porozmawiaj o swoich potrzebach z naszymi specjalistami.
Jakub Orczyk
Członek zarządu / Dyrektor sprzedaży
Zamów bezpłatną konsultację