Digitizing a collection is only half the work. Without descriptive tags, a million scanned photographs are unsearchable, uncategorizable, and effectively invisible — and tagging them by hand does not scale past a few thousand.
We built a pipeline that generates ranked descriptive tags from image content, handwritten annotations, and metadata. In production testing on 35,000 previously unseen images, three researchers completed in two days what had taken a team of ten roughly two weeks. The system has since processed over a million images.
The problem
Institutions digitizing their collections need descriptive tags to sort, search, and curate — to assemble online galleries around a theme, or to let researchers pull everything from a period, region, or subject. Generating those tags by hand across millions of items is prohibitively slow.
The difficulty was less the tagging than the inconsistency of the source material. A single collection spans full colour, sepia, and black-and-white; high-resolution modern scans alongside low-resolution photographs from decades earlier; photographs of paintings and sculptures shot against backing paper that varies in size and colour across the archive; and handwritten annotations in and around the images in shifting ink colours and hands.
Every one of those variations degrades model performance for reasons that have nothing to do with what the image depicts.
What we built
Three stages: standardize the inputs, extract tags from image content, extract tags from text — then merge, deduplicate, and rank.
Standardizing the inputs
We cropped the uniform backing from each image, scaled to a 1000px longest edge preserving aspect ratio, and enforced three colour channels. The cropped backing was kept rather than discarded — that’s where the handwritten annotations live.
Annotation regions were binarized for handwriting recognition: converted to black and white, put through morphological dilation and erosion, then Otsu thresholding.
Tags from image content
Several detection and segmentation models run over the standardized images. Some are off-the-shelf instance segmentation models identifying faces, people, landscapes, forests, water, and mountains. Others we trained on datasets we curated and labelled for targets specific to this collection: sculptures, wall reliefs, and artistic representations of people as distinct from photographs of people.
Outputs are thresholded and combined to establish what is present and what proportion of the frame each element occupies — used later as a proxy for importance.
Tags from text
Binarized annotations run through our handwriting recognition built on CTC decoding. That output combines with image metadata and any written description recorded at capture, then goes through standard NLP cleanup — lowercasing, stop word and punctuation removal, lemmatization.
Keywords are extracted through custom search terms and named entity recognition, pulling people, locations, regions, artists, periods, and subjects.
Merging and ranking
Detected objects and extracted entities are combined, duplicates and redundancies removed, and the surviving tags ranked by importance — image coverage percentage for visual elements, scarcity-based weighting (TF-IDF) for textual ones.
Building a training set for collections that didn’t exist yet
The most consequential decision on this project was about data, not architecture.
The digitized collections available to us were skewed. The overwhelming majority depicted people, sculpture, and locations from North and South America, concentrated in the late 19th and early 20th centuries — an artifact of which collections had been digitized first, not of what the archive contained.
Training on that distribution would have produced excellent measured performance and a system that degraded the moment the client digitized collections from Asia, Europe, or Africa, or from the mid-to-late 20th century when camera technology changed sharply. Good numbers at delivery, deteriorating numbers thereafter.
So we deliberately over-represented image types from collections we knew were coming later, relative to their share of what was then available. We traded measurable performance on the current test set for durability on the collection as it would actually exist.
A second finding, on target scale. Assigning close-up portraits and distant groups of figures to the same class produced models that hallucinated badly — detecting figures in background texture and noise. We sub-segmented classes by target size during training and collapsed them back into their parent classes at inference. The hallucination problem went away and the tags stayed simple.
Results
The pipeline ingests a collection’s images, metadata, and text, generates ranked tags per item, and stores them in a database accessible through a UI we provided.
In primary production testing, 35,000 previously unseen images were processed end to end, then used by three researchers over two days to segment the collection and decide how to fold it into existing galleries.
The previous process for the same work: ten employees for two weeks to generate tags, then another two to three days for researchers to aggregate and segment.
| Previous process | With the pipeline | |
|---|---|---|
| Elapsed time | ~2.5 weeks | 2 days |
| People involved | 10 taggers + researchers | 3 researchers |
| Approximate person-days | ~105 | ~6 |
That is roughly 6× faster in elapsed time and a far larger reduction in labour — the two numbers measure different things and both are real.
Since going into production the system has processed over a million images, sustaining an average 6× speedup against the manual process.
Where the human stays in the loop
The pipeline produces ranked candidate tags. Researchers decide what a collection means — which items belong together, what a gallery is about, what gets published. The system removes the mechanical work of describing a million images individually so curatorial judgment applies to curation.
What it doesn’t do
- The tag vocabulary is deliberately bounded. During refinement we narrowed the achievable tag set to a smaller subset than originally scoped, to be expanded in later model deployments. Shipping a reliable narrow vocabulary beat shipping an unreliable broad one.
- It is not a cataloguing authority. Output is a first pass for researchers, not an authoritative record.
- Handwriting recognition is approximate. CTC decoding returns the closest approximation of the written words; it informs keyword extraction rather than transcribing annotations verbatim.
- Performance depends on the standardization stage. Items that fall outside the expected format degrade downstream.
Stack
TODO — this is the one gap. Name the instance segmentation architectures, the base
for the custom-trained detectors and the size of the labelled set, the NER model or
library, and the storage and UI stack.