AI and Machine Learning in Heritage Documentation: What Actually Works

Why AI Matters — and What It Cannot Yet Do

Heritage documentation involves enormous volumes of image data, repetitive classification tasks, and geometric complexity that exceeds human ability to process efficiently. A thorough photogrammetric survey of a large temple complex produces 5,000–20,000 images. Manual quality review of each image is time-consuming. Condition surveys of large structures produce thousands of annotated photographs. Manual extraction of crack maps from these images takes weeks.

AI addresses the volume problem and the repetition problem. It does not address the interpretation problem. A CNN can classify a 5×5 cm patch of stone surface as 'spalling', 'biological growth', or 'crack' with high accuracy when trained on good data. It cannot tell you whether that spalling is caused by salt crystallisation driven by rising damp, by freeze-thaw, or by historic lime plaster failure — because causation requires contextual knowledge that current AI systems do not have.

Five Proven AI Applications in Heritage

Automated Damage Classification with CNNs

Convolutional neural networks (CNNs) trained on labelled images of heritage surface damage can classify patch-level damage types (cracking, spalling, biological growth, water staining, graffiti, delamination) at speeds and scales impossible for manual annotation.

The standard workflow involves collecting a condition survey photography dataset, patching the images into 256×256 or 512×512 pixel tiles, labelling a training set (typically 1,000–5,000 tiles per class), fine-tuning a pre-trained backbone (ResNet-50, EfficientNet-B4), and applying the trained classifier to the full image dataset. Output is a spatial distribution map of damage types across the surveyed surface.

The critical limitation is dataset domain shift. CNNs trained on European limestone or brick heritage buildings perform poorly on Indian granite, sandstone, or laterite — the stone textures, weathering profiles, and damage types differ significantly. Fine-tuning on India-specific training data is mandatory, not optional, and requires labelled examples from the local building type and stone tradition.

AI-Accelerated Photogrammetry

The most immediately valuable AI integration in heritage documentation is the replacement of classical feature detectors (SIFT, SURF, ORB) with learned feature matchers (SuperPoint, SuperGlue, LOFTR) in the photogrammetric feature matching pipeline.

Classical feature detectors struggle with low-texture surfaces — plain stone walls, monochrome plaster, repetitive carved patterns (often found on Dravidian temple gopurams). Learned matchers trained on a wide variety of surface types find correspondences across viewpoints that SIFT misses, producing denser and more robust sparse point clouds from the same image set.

Agisoft Metashape and RealityCapture have both integrated deep learning matching options that provide significant processing improvement on difficult heritage surfaces with minimal workflow change for the operator. This is the area of AI in heritage documentation with the clearest, most immediate return on adoption — it requires no new hardware beyond what photogrammetric processing already demands.

NeRF and Gaussian Splatting for Heritage Geometry

Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) are neural rendering techniques that reconstruct a 3D scene from a set of 2D images by learning an implicit function that predicts the appearance of the scene from any viewpoint.

For heritage documentation, these techniques have two compelling advantages over traditional photogrammetry: they handle reflective, translucent, and glossy surfaces (polished stone, gilded surfaces, glazed tiles) that confuse traditional SfM feature matchers; and they produce photorealistic novel views from any angle, making them powerful for heritage visualisation and interpretation for non-specialist audiences.

The limitations are equally important. NeRF and Gaussian Splatting output is not directly metric. The reconstructed geometry is excellent for visualisation but requires careful validation before use for architectural measurement, condition mapping, or regulatory submissions. Processing a 500-image heritage dataset with Instant-NGP or nerfstudio on a local GPU takes 30 minutes to 4 hours depending on GPU and scene complexity — far faster than early NeRF implementations but still significant.

Decision Table: Traditional vs AI-Assisted vs NeRF

Use this table when deciding which processing approach to use for a given heritage documentation task. The choice is not always 'most advanced' — it is always 'most appropriate for the deliverable required'.

ML-Based Change Detection

Machine learning change detection compares photogrammetric models or orthophotos captured at different dates to identify surface change — new damage, material loss, graffiti, unauthorised repair. The ML component improves on simple pixel differencing by distinguishing genuine change from lighting variation, vegetation movement, and processing artefacts.

The most practical implementation for heritage sites uses bitemporal orthophoto comparison with a trained Siamese network or change detection CNN. Input is a pair of registered orthophotos from two survey dates; output is a change probability map highlighting areas where surface appearance has changed beyond a trained threshold.

This approach requires a consistent image capture protocol between surveys — same flight altitude, similar lighting conditions, same camera settings — to minimise false positives from capture variation. Without this consistency, the ML classifier spends most of its capacity distinguishing acquisition differences from genuine change, reducing sensitivity for real damage detection.

Hardware Requirements and GPU Costs in India

AI heritage documentation workflows have real hardware requirements that affect project economics. The following guidance covers both local (owned hardware) and cloud (rented GPU) options relevant to Indian practitioners.

Indian Context: Research, Datasets and Constraints

**Academic research:** IIT Bombay, IIT Madras, NIT Trichy, and CEPT University Ahmedabad have active research groups working on AI in heritage documentation. IIT Bombay has published work on automated Dravidian temple element recognition and point cloud segmentation of Indian heritage structures. IIT Madras has research in heritage building energy modelling with AI components. However, peer-reviewed heritage-specific AI tools from Indian institutions have not yet reached the same software maturity as European and American research outputs.

**Training dataset gap:** The most significant AI constraint in Indian heritage work is the absence of large, labelled training datasets for Indian stonework damage types, regional architectural elements, and the specific weathering profiles of laterite, granite, sandstone, and lime plaster in tropical conditions. Existing open heritage AI datasets are dominated by European limestone and brick — effectively unusable without significant fine-tuning for Indian applications.

**ISRO and national programmes:** ISRO's AI research activities focus on satellite image analysis, including some heritage-adjacent applications (archaeological site detection, change detection). The National Mission for Manuscripts (NaMaMi) and the National Mission on Cultural Mapping have digital data components, but AI integration is nascent.

**Practical reality for practitioners:** Most heritage professionals in India using AI documentation tools are doing so through commercial photogrammetry software (Metashape, RealityCapture) that has integrated AI features — they are AI users, not AI developers. Custom CNN training for damage classification is being done by a small number of practitioners and university groups. NeRF tools (nerfstudio, instant-ngp) are being used in urban architecture offices and architectural schools more than heritage conservation organisations.

Field Notes from Practice

**NeRF on a Dravidian gopuram: the good and the frustrating:** NeRF reconstruction of a multi-storey gopuram from approximately 800 drone images produced a photorealistic 3D model in under two hours of GPU processing — geometrically faithful enough for public interpretation and stakeholder presentations. The same dataset processed through Metashape produced a dense point cloud metric enough for measured drawing extraction. The two products served completely different purposes and both were genuinely useful. The frustration: NeRF output in any format exportable to CAD or BIM required significant post-processing — the pipeline from NeRF to metric CAD deliverable remains a workflow bottleneck.

**CNN damage classifier failure on laterite:** A CNN trained on European stone heritage and applied (without fine-tuning) to laterite block surfaces in Tamil Nadu misclassified natural iron oxide ferricrete inclusions and weathering pits as damage at a rate exceeding 40%. The tool was doing what it was trained to do — recognising textural anomalies — but the model had no knowledge that these anomalies were natural features of laterite, not deterioration. Fine-tuning on 2,000 labelled laterite patches reduced the false positive rate to under 8%. The lesson: pre-trained models on non-Indian stone always require fine-tuning before production use on Indian heritage.

**The quality-of-input ceiling:** Every AI acceleration tool hits the same ceiling: AI cannot recover information that was not captured. A photogrammetric dataset with motion blur, insufficient overlap, and poor lighting in shadow zones produces a poor sparse point cloud regardless of whether SIFT or SuperGlue is used for feature matching. The most consistent lesson from deploying AI tools in heritage documentation is that image capture quality remains the primary determinant of output quality, not the AI tool selected.

Common Mistakes Made by Professionals

• Applying AI damage classifiers trained on European stone datasets to Indian heritage without fine-tuning: produces unacceptable false positive rates on laterite, granite, and sandstone surfaces with natural textural variation.
• Treating NeRF and Gaussian Splatting output as metric without validation: visually compelling NeRF models can have local geometric distortions of 5–50 mm relative to ground truth. Always validate against check points or a parallel SfM survey before using for measurement.
• Using AI-generated condition maps without expert review: AI classifiers produce probability maps, not conservation assessments. A heritage-qualified professional must interpret the output before it becomes a conservation recommendation.
• Over-specifying hardware before testing workflows: buying a high-end GPU workstation before validating that AI tools actually improve output quality for the specific heritage type and workflow is a common and expensive mistake. Test on cloud GPU first.
• Failing to log training data provenance: a CNN model is only as reliable as its training data. Projects that do not document which images were used for training, who labelled them, and what inter-labeller agreement was achieved cannot validate model reliability or reproduce results.
• Expecting AI to automate the architectural judgment step: automated tools can classify, segment, and detect. They cannot determine which cracks are structurally significant, which material is historically authentic, or which intervention strategy is appropriate. These steps remain human.

Frequently Overlooked Considerations

• The cost of building training data: labelling 5,000 image patches for a CNN damage classifier takes 40–80 hours of skilled annotation time. This cost is rarely included in AI tool adoption budgets and is often the true limiting factor rather than GPU cost.
• Model drift over time: a CNN trained in 2024 on a specific site's imagery will perform differently on 2027 imagery of the same site — because weathering, cleaning, and repair change the surface's visual appearance. Models require periodic retraining or accuracy monitoring.
• Licensing of AI tools in conservation practice: some AI tools in heritage documentation are research code under academic licenses that prohibit commercial use. Confirm licensing terms before using any open-source AI tool in a client project.
• Energy consumption of large model training: training a CNN on a local GPU for 8 hours consumes significant electricity. For remote heritage sites running on solar power, AI processing should be reserved for cloud platforms rather than on-site hardware.
• Integration with existing heritage records systems: AI outputs are most valuable when they are linked to the spatial heritage record (GIS, HBIM). Outputs that exist only as standalone images or standalone GeoJSON files are quickly disconnected from the active conservation management workflow.
• The hallucination problem in LLM-assisted heritage research: large language models (LLMs) used to assist with heritage history research and significance assessment can confidently produce incorrect historical facts — dates, builder attributions, construction techniques — that are not in their training data. Verify every specific historical claim from an LLM against primary sources.

AI Tool Evaluation Checklist

Before adopting any AI tool for a heritage documentation project, answer these questions:

1. 1What specific task does this tool perform — and does the existing non-AI workflow have a genuine bottleneck at that task?
2. 2What training data was the model trained on — is it representative of the stone type, architectural tradition, and weathering profile of this specific site?
3. 3What is the validation accuracy on comparable Indian heritage imagery — not the accuracy reported in the vendor's European case studies?
4. 4What is the output format — can it integrate with the existing documentation workflow (GIS, Metashape project, Revit model)?
5. 5What hardware does it require — is that hardware available locally or via cloud within project budget?
6. 6Is the tool licensed for commercial use in client projects?
7. 7Is a heritage-qualified professional available to review AI output before it becomes a conservation recommendation or regulatory submission?
8. 8Is the processing pipeline fully documented so results can be reproduced and independently validated?
9. 9What is the failure mode — how does the tool behave on inputs outside its training distribution, and is that failure visible (low confidence score) or silent (high-confidence wrong prediction)?
10. 10Who maintains the model — if it is research code from a university, what happens when the researchers move on?