Skip to content

OpenConceptLab/ocl_issues#3012 #3013 | $clone keeps mappings to other code systems and points copied mappings source URLs at their concepts sources; repair command for HEAD mappings - #995

Open
snyaggarwal wants to merge 7 commits into
masterfrom
issues#3012
Open

snyaggarwal wants to merge 7 commits into
masterfrom
issues#3012

Conversation

@snyaggarwal

@snyaggarwal snyaggarwal commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Fixes OpenConceptLab/ocl_issues#3012 and OpenConceptLab/ocl_issues#3013.

Why one PR: both bugs are in the same few lines of the $clone path (Source.clone_resources → Mapping.clone → Source.clone_mappings). #3013 decides which concept a copied mapping points at, and #3012 makes the source fields follow that concept. Fixed apart, the second PR would rewrite the same lines and conflict. The #3012 repair also has to know about #3013's mappings: on those, to_source_url is the only record of the external code system they used to point at, and repairing #3012 alone would overwrite it.

#3012: copied mappings keep the original's source URLs

Mapping.clone() copied the original's from_source_url/to_source_url and source versions even when it swapped in a concept from the target source, and clone_mappings() updated only the source IDs. So a copied mapping's from_source_url still named the source it was cloned from, and a mapping pointed at an equivalent concept kept the old to_source_url. The equivalency mappings $clone creates had no source URLs at all.

  • Mapping.clone(user, from_concept, to_concept): when a substituted concept is in a different source from the original mapping's concept, the copy's source ID and URL are set to that concept's source, and the source version (which belonged to the old source) is cleared. The check compares the concepts' sources, not the stored source ID, so older rows with no to_source_id keep their external URL (for example a canonical URL) and pinned version. The from and to sides share one helper, Mapping.get_source_fields_from_concept.
  • Source.clone_mappings: fills a missing source ID or URL from the linked concept and keeps a recorded URL and version. The by-mnemonic re-resolution of unsaved concepts is removed. Its only caller, clone_resources, always passes saved concepts, so only tests used it.

#3013: mappings to other code systems were retargeted

clone_resources called get_equivalent_concept() for every copied mapping. If the target source had a SAME-AS to the mapping's target, the copy was pointed at that concept even when the target was LOINC or ICD-10, and the external code was lost.

  • Only targets in the source being cloned from (Q-AND-A answers, set members) are moved to their equivalent in the target source. Mappings to other code systems are copied unchanged, keeping their concept, code and source.
  • A concept's self-mapping (e.g. SAME-AS CIEL:136713 on CIEL 136713) still becomes a self-mapping on the new concept. The original is self-referencing too, so that's the faithful copy, and with #3012 its to_source_url now matches.
  • The cascade concepts are loaded with select_related('parent'), the cascade mappings with select_related('from_concept', 'to_concept'), and the equivalent concept with its source. So the source check, Mapping.clone and the URL fill add no queries per mapping.

Repair command: repair_mapping_source_urls

As asked in the #3012 comments. Use --dry-run to list only.

  • HEAD mappings only (the versioned objects, id = versioned_object_id). Mapping versions are not changed, so released source versions and their exports stay as they were. A mapping's versions list and version detail, and collection expansions that hold a version, also keep the old URL. The next edit of a mapping creates its new version from the repaired HEAD.
  • For each HEAD mapping with a from/to source set, it rewrites the URL to that source's URI when the URL is missing (NULL or empty, including on sources whose canonical URL is blank), or is the URI or canonical URL of a different source. In the second case it also clears the source version. URLs that match the mapping's own source (URI or canonical) or no known source are left alone.
  • #3013's mappings are listed, not repaired. If a mapping's to_concept is in its own source but to_source_url names a source other than its from_source_url (the source it was cloned from), its external target was replaced. The two URLs are compared by the source they name, URI or canonical URL. The command lists each such mapping with both URLs so it can be fixed by hand, and leaves its to_source_url in place. The same run then repairs from_source_url, so this output is the record of the cloned-from source. The to side runs before the from side so this comparison sees the original from_source_url, and a later run still skips these mappings. Retargeted answers, whose to_source_url names the cloned-from source, are repaired as usual. The command doesn't restore the external codes: the copy no longer has them, and #3013 plans a hand fix.
  • It finds only the sources that have such mappings, repairs each in its own transaction, and prints a count per source. Source URLs aren't in the mapping index or checksums, so no re-index is needed.

Test changes

Two tests from #989 (cloned_mapping_in_local_source in the importer tests and test_put_200_keeps_the_local_link_of_a_cloned_mapping) relied on $clone leaving the origin's to_source_url. They now set that state explicitly after the clone, so the edit and import behavior for mappings cloned before this fix stays covered. test_clone_mappings_resolves_missing_concepts_by_mnemonic is removed along with the re-resolution.

New tests:

  • Clone with an equivalency map type: the copied mapping and the equivalency mapping get the target's from_source_url with no version, the equivalency mapping's to_source_url is the original source, and the external side keeps its URL and version.
  • Clone of an older mapping with no to_source_id and a canonical to_source_url: the URL and pinned version are kept.
  • Clone that points an in-source answer at its equivalent: to_source_url is the target and its version is cleared.
  • Clone of a BROADER-THAN mapping to an external concept that the target has a SAME-AS to: the copy still points at the external concept and source. This test fails without the fix.
  • Clone of a concept with a SAME-AS to itself: the copy is a self-mapping on the cloned concept, with the target's source URLs.
  • End to end through POST .../concepts/$clone/ with mapTypes=Q-AND-A,SAME-AS and equivalencyMapType=SAME-AS (a staff user, so no clone budget): the Q-AND-A answer moves to its existing equivalent with the target's source URLs, and the SAME-AS to an external concept keeps its concept, code and source, even though the target has a SAME-AS to that concept. This test fails without the #3013 fix.
  • The repair command:
    • A dry run changes nothing.
    • Stale URI, stale canonical, NULL and empty URLs are fixed, including an empty URL on a source with a blank canonical URL. That case fails without the fix.
    • Retargeted answers are fixed, including one whose two URLs name the origin by URI and by canonical URL.
    • Own-canonical and unknown URLs are kept.
    • A replaced external target is listed with both URLs and its to_source_url kept.
    • A mapping version with a stale URL is left unchanged.
    • A second run repairs nothing, still lists the replaced target and still keeps its to_source_url.

pylint -j0 core/ exits 0. Locally, the source, mapping, concept, bundle, importer, source API, mapping API and cross-repo access tests pass (987). The one failure, test_grandfathered_npm_url_is_downloaded_and_handed_over_as_upload, fails the same on master.

🤖 Generated with Claude Code

…ource_url/to_source_url at the mapped concepts sources; repair command for existing mappings
Comment thread core/sources/models.py Resolved
Comment thread core/sources/models.py Resolved
Comment thread core/sources/models.py Resolved
@snyaggarwal
snyaggarwal requested a review from paynejd November 16, 2026 09:06
@snyaggarwal snyaggarwal self-assigned this Nov 16, 2026
…a substituted concept, legacy rows keep their external URL, repair covers HEAD mappings only, one transaction per source
@snyaggarwal snyaggarwal changed the title OpenConceptLab/ocl_issues#3012 | $clone points copied mappings from_source_url/to_source_url at the mapped concepts sources; repair command for existing mappings OpenConceptLab/ocl_issues#3012 | $clone points copied mappings from_source_url/to_source_url at the mapped concepts sources; repair command for HEAD mappings Nov 16, 2026
@snyaggarwal
snyaggarwal marked this pull request as ready for review November 16, 2026 09:23
Comment thread core/mappings/models.py Resolved
Comment thread core/sources/models.py Resolved
Comment thread core/mappings/models.py Resolved
…rce being cloned from to their equivalent; repair lists mappings whose external target was replaced instead of overwriting their URL
@snyaggarwal snyaggarwal changed the title OpenConceptLab/ocl_issues#3012 | $clone points copied mappings from_source_url/to_source_url at the mapped concepts sources; repair command for HEAD mappings OpenConceptLab/ocl_issues#3012 #3013 | $clone keeps mappings to other code systems and points copied mappings source URLs at their concepts sources; repair command for HEAD mappings Nov 16, 2026
Comment thread core/sources/models.py Resolved
… target and matches the cloned-from source by URI or canonical URL; rerun and self-mapping tests
Comment thread core/mappings/models.py Resolved
Comment thread core/mappings/tests/tests.py Resolved
…ith a blank canonical URL; clone loads concept sources up front
Comment thread core/sources/tests/tests.py Resolved

@snyaggarwal snyaggarwal left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review verdict: ready to merge (one non-blocking suggestion)

Reviewed at c73da962 (fifth pass). Every finding from the four earlier passes is fixed, and each thread has a reply.

#3012 (source URLs on copied mappings): correct. Mapping.clone sets the source from a substituted concept, comparing the concepts' sources rather than the stored source ID, so older rows without to_source_id keep their external URL and pinned version. clone_mappings only fills missing fields. The from and to sides share one helper.

#3013 (external targets retargeted): correct. Only targets in the source being cloned from are moved to their equivalent. $clone always cascades within one source (or one of its versions), so the same-source check matches the cascade. Self-mappings stay self-mappings, which matches the original.

Repair command: safe to run, after a --dry-run first.

  • It changes HEAD rows only, so released versions and their exports stay as they were.
  • It finds only the sources that need repair and repairs each in its own transaction.
  • The #3013 mappings are listed with both URLs and their external to_source_url is kept. The to side runs first, and a second run still skips them.
  • Source URLs aren't indexed or part of checksums, so no re-index is needed.

Before running the repair in production: run --dry-run and save the list of skipped mappings. It's the record for the #3013 hand fix, because the run then repairs those mappings' from_source_url.

Open (non-blocking): one inline suggestion for an integration test through $clone.

Checks: pylint -j0 core/ exits 0. 987 tests pass locally across the source, mapping, concept, bundle, importer and API suites. The one failure, test_grandfathered_npm_url_is_downloaded_and_handed_over_as_upload, fails the same on master.

…ir equivalent, targets in other code systems are kept
self.assertEqual(external_copy.to_concept_id, external_concept.id)
self.assertEqual(external_copy.to_concept_code, external_concept.mnemonic)
self.assertEqual(external_copy.to_source_url, external.uri)
self.assertFalse(copies.filter(to_concept=existing_external).exists())

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Style (nit, non-blocking): add a second blank line after this test.

There's only one blank line between the end of this test and class SourceVersionsChangelogOutputViewTest, while PEP 8 (E302) and the rest of the file use two. Pylint, the only CI linter, doesn't flag it, so CI passes. But flake8 or an editor's formatter would, and a later formatting pass would add an unrelated hunk to this file.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 2643c6b: two blank lines before class SourceVersionsChangelogOutputViewTest.

@snyaggarwal snyaggarwal left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review verdict: ready to merge (one whitespace nit)

Reviewed at 4285451b (sixth pass). Every finding from the earlier passes is fixed, and each thread has a reply. The only change since the last verdict is the end-to-end $clone test, which that verdict suggested.

The new test posts a real $clone (nothing mocked) with mapTypes=Q-AND-A,SAME-AS and equivalencyMapType=SAME-AS, as a staff user so there's no clone budget. It covers the whole path, SourceConceptsCloneView → Bundle.clone → clone_with_cascade → Concept.cascade → clone_resources, and checks:

  • The Q-AND-A answer moves to its existing equivalent, with the target's source URLs.
  • The SAME-AS to an external concept keeps its concept, code and source, even though the target has a SAME-AS to that concept.

It fails without the #3013 fix. Its from_source_url=target.uri filter picks out the clone's own mappings, not the pre-existing ones.

Unchanged from the last verdict:

  • #3012: copied mappings take the source of the concept they point at, and older rows without to_source_id keep their external URL and version.
  • #3013: only targets in the source being cloned from are moved to their equivalent.
  • Repair command: it changes HEAD rows only, finds only the sources that need repair, and repairs each in its own transaction. The #3013 mappings are listed with both URLs and their external to_source_url is kept, again on a second run. Empty URLs are handled, including on sources with a blank canonical URL.

Before running the repair in production: run --dry-run and save the list of skipped mappings. It's the record for the #3013 hand fix, because the run then repairs those mappings' from_source_url.

Open (non-blocking): a missing blank line after the new test (inline).

Checks: pylint -j0 core/ exits 0 locally. 987 tests passed locally at c73da962, and the clone API tests (9) pass at 4285451b. The one failure, test_grandfathered_npm_url_is_downloaded_and_handed_over_as_upload, fails the same on master. CI on 4285451b is still pending (Pylint queued).

@snyaggarwal snyaggarwal left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review verdict: ready to merge (no findings)

Reviewed at 2643c6b1 (seventh pass). The only change since the last pass adds the missing blank line after the $clone API test (PEP 8 E302). I also re-read the whole diff, including the test-file changes, and found nothing new. Every earlier finding is fixed, and each thread has a reply.

Summary of the PR as it stands:

  • #3012: copied mappings take the source ID, URL and (cleared) version of the concept they now point at. Older rows without to_source_id keep their external URL and pinned version. Equivalency mappings get their source URLs.
  • #3013: only targets in the source being cloned from are moved to their equivalent in the target. Mappings to other code systems are copied unchanged, and a self-mapping stays a self-mapping.
  • repair_mapping_source_urls:
    • It changes HEAD rows only, finds only the sources that need repair, and repairs each in its own transaction.
    • Missing URLs (NULL or empty, including on sources with a blank canonical URL) and URLs naming another source are fixed.
    • The #3013 mappings are listed with both URLs and their external to_source_url is kept, again on a second run.
  • Tests: model-level clone tests, a repair-command test (dry run, a real run, and a second run), and an end-to-end $clone API test. The #3013 and blank-canonical cases fail without their fixes.

Before running the repair in production: run --dry-run and save the list of skipped mappings. It's the record for the #3013 hand fix, because the run then repairs those mappings' from_source_url.

Checks: pylint -j0 core/ exits 0 locally. 987 tests passed locally at c73da962, and the clone API tests passed at 4285451b. Since then only that test and this whitespace line changed. The one failure, test_grandfathered_npm_url_is_downloaded_and_handed_over_as_upload, fails the same on master. CI on 2643c6b1: Snyk passed, Pylint is still queued.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: $clone leaves from_source_url (and to_source_url) pointing at the original source on copied mappings

1 participant