A dataset and toolchain built from the Script Chart Syria – Jordan spreadsheet: five inscriptions, 28 letters, four positional forms, tied back to the plates they were cut from.
The chart is built: chart.html is the site. It remains a research work in
progress: the readings are provisional and the third-party plate images are not
licensed for redistribution.
The tools read JSON out of data/, which file:// blocks. Serve the folder:
python3 -m venv .venv
.venv/bin/pip install -r requirements.txt
./serve.sh # http://localhost:8123/Before committing a change, run the repository consistency check:
.venv/bin/python scripts/check_repo.pychart.html the site
index.html landing page + corpus summary
tools/annotate.html two-pane glyph tagger
tools/align.html tie each tracing to its photograph (finished)
assets/plates/ the photographs and tracings
assets/glyphs/ 95 glyph crops extracted from the spreadsheet
assets/chart/ the crops the site shows, cut from our own plates
data/glyphs.json the tagged corpus — the source of truth
data/chart-built.json generated from it by build_crops.py
data/chart.json the spreadsheet's cells: inscriptions x letters x positions
data/alignments.json tracing -> photo homographies (from align.html)
data/seeds.json template-matched crop locations (build-stage residue)
data/annotations.json polygons from the old annotator (build-stage residue)
scripts/ the extraction + matching pipeline
data/glyphs.json is the only hand-authored data file; assets/chart/ and
data/chart-built.json are generated from it. The last two entries are
superseded and read by nothing — kept for provenance.
The spreadsheet is a Google Sheets export, and every glyph is an in-cell
image. Each one carries a <xdr:col>/<xdr:row> anchor in
xl/drawings/drawing1.xml, and that anchor is the dataset — column gives the
inscription and positional form, row gives the letter. scripts/extract_chart.py
walks those anchors, writes each image to assets/glyphs/ under a
{inscription}.{letter}.{position}.png name, and emits data/chart.json.
95 glyph images across 93 cells (Jabal Ramm h final and s medial each carry
two variants, written as …-2.png and addressed as …#2).
| Inscription | Glyphs | Plates |
|---|---|---|
| Jabal Ramm (Nabataeo-Arabic) | 15 | photo, tracing |
| Jabal Says | 23 | photo, tracing |
| Ḥarrān | 26 | photo (zoom + wide + composite), tracing |
| Zebed | 14 | photo |
| Ḥimà-Sud PalAr 8 | 17 | photo, tracing |
Ḥarrān's wide shot is kept because it carries a word at far left that the zoom crops off; the two are merged into a composite plate (below), which is what Ḥarrān is actually worked on. Zebed has no tracing, so its glyphs are annotated on the photo directly.
Alignment is done for all five jobs — data/alignments.json carries a
homography per plate pair, fitted from 6–20 point pairs each:
| Job | Pairs | Mean err | Worst |
|---|---|---|---|
| jabal-ramm | 6 | 1.2 px | 2.7 px |
| jabal-says | 17 | 5.1 px | 15.6 px |
| harran | 20 | 11.0 px | 22.5 px |
| harran-wide | 11 | 3.2 px | 10.4 px |
| hima-sud | 13 | 6.3 px | 19.9 px |
Ḥarrān's larger residual is the stone itself — the surface is not planar, which is the ceiling for a single homography. It is still well within glyph size.
Tagging is done: 112 glyphs across 90 cells, each carrying a box on the photograph and a box on the tracing, and so producing both pictures of itself.
| Inscription | Glyphs tagged |
|---|---|
| Jabal Says | 33 |
| Ḥarrān | 28 |
| Jabal Ramm | 22 |
| Ḥimà-Sud | 16 |
| Zebed | 13 |
These are our own boxes on our own plates, not the spreadsheet's crops — the spreadsheet only ever served as a placement guide. The two therefore differ, in both directions, and reconciling them is the outstanding data task:
.venv/bin/python scripts/sheet_diff.pyIt reports cells the spreadsheet has that we don't, cells we have that it doesn't, and any non-connector wrongly filed as initial or medial (currently none). Ḥimà-Sud is expected to diverge — its spreadsheet readings were not trusted and it was re-derived from scratch — so it is reported separately. The output of that command is the current review list.
Some cells cannot be placed at all: their crops came from photographs that are not in this repo — see Missing sources below.
Ḥarrān is worked on harran.photo-composite.jpg, a single plate carrying the
best available detail everywhere: the zoom photograph at 1:1 where it reaches,
and the wide photograph upscaled to fill in around it — which is what restores
the Greek opening and the left-hand word the zoom crops off.
scripts/harran_composite.py builds it. The one thing worth knowing is how the
zoom is placed on the wide. Composing the two tracing homographies
(H_tracing→wide @ inv(H_tracing→zoom)) compounds both of their residuals, and
Ḥarrān's tracing fit already carries 11px of non-planar stone; the seam that
produces is visibly stepped, with the mouldings failing to meet. Estimating the
mapping directly between the two photographs with SIFT + RANSAC fixes it —
gradient NCC over the overlap goes from -0.00, meaning no geometric agreement at
all, to 0.15, and the mouldings then run continuously across the seam. The
composed homography is kept as a sanity check: if the direct estimate drifts
more than 120px from it, the script falls back rather than trusting it.
The frame is the wide plate's extent at 4.29× — the zoom's resolution advantage — so the zoom resamples 1:1 and loses nothing. The fill is exposure-matched over the overlap and feathered over 150px. The feather is deliberately moderate: the fill is a 4.3× upscale and where that starts should stay visible rather than being blended into looking like real detail.
Two caveats:
- The annotations are exact. They were moved onto the composite by the real
pixel mappings, and each retains a
sourcePlaterecording where it was drawn. - The tracing overlay on the composite is the compromise. Refitting it from
all 31 point pairs of both Ḥarrān jobs gives 19px mean over the zoom region
and 27px over the wide-only region. Routing it through the zoom job instead
would give 11px over the zoom but 100px over the wide-only region — unusable
exactly where the extra word is — so the even fit wins. Repick
Ḥarrān — compositeinalign.htmlto improve on it.
The crops were cut out of these plates, so scripts/seed_boxes.py finds most of
them again by multi-scale normalised cross-correlation (CLAHE-equalised, so a
crop matches its plate despite lighting differences), searching every plate
belonging to that inscription. Misses get a second pass that also sweeps
rotation, guarded by a minimum match width so it can't "solve" the problem by
shrinking the template until noise correlates.
.venv/bin/pip install -r requirements.txt
.venv/bin/python scripts/extract_chart.py
.venv/bin/python scripts/seed_boxes.py # a few minutesEach hit lands in data/seeds.json as a box on a named plate, which the
annotator pre-places — so labelling is mostly review and nudge rather than
drawing 95 polygons cold. Anything below the confidence threshold is marked
placed: false and shows up in the annotator's queue as "no seed".
Worth knowing: most crops came from the photographs, not the tracings.
scripts/accept_seeds.py promotes reviewed seeds into data/annotations.json
as confirmed polygons. Re-running merges rather than overwrites — anything
already in the file wins, because it has been edited in the tool since.
chart.html is the finished view. It opens on Every form — the whole corpus
on one sheet, letters down the side, joining position across, every attested
form in its cell — with a tab per inscription after it. Grouped by inscription,
each text keeps the same sub-row inside a letter block, so reading across a row
compares positions within one hand and reading down compares hands.
Identity there is carried by row position and a label rather than by colour. A five-way categorical palette would have to clear real separation thresholds and would still fight a warm paper-and-ink page, where the marks are photographs of stone and any hue applied over them competes with the evidence.
Readings will need changing. scripts/relabel.py is the operation:
.venv/bin/python scripts/relabel.py jabal-ramm-5 waw # letter only
.venv/bin/python scripts/relabel.py jabal-ramm-5 waw final # letter + position
.venv/bin/python scripts/build_crops.py # filenames encode bothIt takes a letter id (waw) or its transliteration (w), clears the fromSeed
flag since the assignment is no longer the matcher's, and reports when the
change leaves an inscription with more than one glyph in that cell.
The inscriptions are ancient; the photographs and facsimiles of them are not.
Each is an original work, and the rights recorded in DiCoNab are carried through
CREDITS in chart.html to a line beside every image on the page:
| Plate | Rights |
|---|---|
| jabal-ramm.tracing.jpg | Public domain |
| jabal-ramm.photo.jpg | © G.L. Harding |
| zebed.photo.jpg | © A. Grohmann |
| jabal-says.photo.jpg | © M.C.A. Macdonald |
| jabal-says.tracing.jpg | © M. Gorea |
| harran.photo-zoom.jpg, harran.photo-wide.jpg | © C.J. Robin |
| harran.photo-composite.jpg | © C.J. Robin, derived from the two above |
| harran.tracing.jpg | © M. Gorea |
| hima-sud.photo.jpg | © C. Robin, MAFSN |
| hima-sud.tracing.jpg | © L. Nehmé |
Credit is not a licence. DiCoNab displaying these with a credit line does not extend permission to anyone else — it identifies who to ask. Four of the holders are living scholars; two (Harding, Grohmann) are not, which is the harder case. Permission has not been sought yet, and should be before this is published anywhere public.
Five unplaced cells (setting Ḥimà-Sud aside) are not a matching failure; their crops were cut from photographs that are not in this repo. Each is visibly a different medium from the plate it belongs to:
- Zebed (4) — colour photographs of a grey-brown stone, where the Zebed plate here is a black-and-white halftone print.
- Jabal Ramm (1) — likewise colour, against a halftone plate.
Intensity matching and an edge-gradient matcher both bottom out around 0.2–0.4
on these, well under the 0.62 threshold, which is what you would expect when the
source image simply is not present. Adding the right plate and re-running
seed_boxes.py is the fix; otherwise they are drawn by hand.
Click the same feature in the tracing and in the photograph to make a point pair; four or more pairs solve a homography (least squares, so extra pairs improve the fit rather than being ignored). The Tracing slider warps the tracing over the photo with that homography so you can check the fit directly, and the panel reports mean and worst-point reprojection error.
Export writes data/alignments.json. Import reads it back, so alignments move
between machines.
Tracing on the left, photograph on the right, synced through that inscription's homography.
Drag a box round a letter in either pane and the other pane gets the same box mapped across. Click its letter and position from the palette (1–4 also set position). That one action produces both pictures of that glyph.
The two boxes are independent once drawn. The alignment is only good to a few pixels and individual letters drift, so when the tracing box sits slightly off you just drag its corners — the photo box is untouched. Tracing ← photo re-derives it from scratch if you want to start over.
The right-hand panel previews exactly what will be cut, and the Chart reference below it shows the spreadsheet's crop for that cell — a placement guide only, never the image that ships.
The left-hand coverage grid is the chart itself filling in: 28 letters × 4 positions, green where you have a glyph. Clicking an empty cell assigns the current box to it.
Work autosaves to localStorage; Export JSON writes data/glyphs.json.
.venv/bin/python scripts/build_crops.pyCuts every tagged box out of your own plates into assets/chart/ as
{inscription}.{letter}.{position}.{tracing,photo}.png, and writes
data/chart-built.json recording each glyph's images and its location on the
full plate, which is what a hover-to-locate view needs.
data/annotations.json:
{"annotations": [
{"id": "harran.meem.initial", "inscription": "harran", "letter": "meem",
"position": "initial", "plate": "harran.photo-zoom.jpg", "confirmed": true,
"extra": false, "points": [[1204, 611], [1268, 611], [1268, 668], [1204, 668]]}
]}Points are pixel coordinates in the named plate. To put a polygon drawn on a
tracing onto its photograph, apply that job's homography from
data/alignments.json; H is row-major 3×3 flattened, mapping source plate to
target plate.