Repository navigation
OpenConceptLab/ocl_issues#3012 #3013 | $clone keeps mappings to other code systems and points copied mappings source URLs at their concepts sources; repair command for HEAD mappings - #995
Conversation
…ource_url/to_source_url at the mapped concepts sources; repair command for existing mappings
…a substituted concept, legacy rows keep their external URL, repair covers HEAD mappings only, one transaction per source
…rce being cloned from to their equivalent; repair lists mappings whose external target was replaced instead of overwriting their URL
… target and matches the cloned-from source by URI or canonical URL; rerun and self-mapping tests
…ith a blank canonical URL; clone loads concept sources up front
snyaggarwal
left a comment
There was a problem hiding this comment.
Review verdict: ready to merge (one non-blocking suggestion)
Reviewed at c73da962 (fifth pass). Every finding from the four earlier passes is fixed, and each thread has a reply.
#3012 (source URLs on copied mappings): correct. Mapping.clone sets the source from a substituted concept, comparing the concepts' sources rather than the stored source ID, so older rows without to_source_id keep their external URL and pinned version. clone_mappings only fills missing fields. The from and to sides share one helper.
#3013 (external targets retargeted): correct. Only targets in the source being cloned from are moved to their equivalent. $clone always cascades within one source (or one of its versions), so the same-source check matches the cascade. Self-mappings stay self-mappings, which matches the original.
Repair command: safe to run, after a --dry-run first.
- It changes HEAD rows only, so released versions and their exports stay as they were.
- It finds only the sources that need repair and repairs each in its own transaction.
- The #3013 mappings are listed with both URLs and their external
to_source_urlis kept. The to side runs first, and a second run still skips them. - Source URLs aren't indexed or part of checksums, so no re-index is needed.
Before running the repair in production: run --dry-run and save the list of skipped mappings. It's the record for the #3013 hand fix, because the run then repairs those mappings' from_source_url.
Open (non-blocking): one inline suggestion for an integration test through $clone.
Checks: pylint -j0 core/ exits 0. 987 tests pass locally across the source, mapping, concept, bundle, importer and API suites. The one failure, test_grandfathered_npm_url_is_downloaded_and_handed_over_as_upload, fails the same on master.
…ir equivalent, targets in other code systems are kept
| self.assertEqual(external_copy.to_concept_id, external_concept.id) | ||
| self.assertEqual(external_copy.to_concept_code, external_concept.mnemonic) | ||
| self.assertEqual(external_copy.to_source_url, external.uri) | ||
| self.assertFalse(copies.filter(to_concept=existing_external).exists()) |
There was a problem hiding this comment.
Style (nit, non-blocking): add a second blank line after this test.
There's only one blank line between the end of this test and class SourceVersionsChangelogOutputViewTest, while PEP 8 (E302) and the rest of the file use two. Pylint, the only CI linter, doesn't flag it, so CI passes. But flake8 or an editor's formatter would, and a later formatting pass would add an unrelated hunk to this file.
There was a problem hiding this comment.
Fixed in 2643c6b: two blank lines before class SourceVersionsChangelogOutputViewTest.
snyaggarwal
left a comment
There was a problem hiding this comment.
Review verdict: ready to merge (one whitespace nit)
Reviewed at 4285451b (sixth pass). Every finding from the earlier passes is fixed, and each thread has a reply. The only change since the last verdict is the end-to-end $clone test, which that verdict suggested.
The new test posts a real $clone (nothing mocked) with mapTypes=Q-AND-A,SAME-AS and equivalencyMapType=SAME-AS, as a staff user so there's no clone budget. It covers the whole path, SourceConceptsCloneView → Bundle.clone → clone_with_cascade → Concept.cascade → clone_resources, and checks:
- The Q-AND-A answer moves to its existing equivalent, with the target's source URLs.
- The
SAME-ASto an external concept keeps its concept, code and source, even though the target has aSAME-ASto that concept.
It fails without the #3013 fix. Its from_source_url=target.uri filter picks out the clone's own mappings, not the pre-existing ones.
Unchanged from the last verdict:
- #3012: copied mappings take the source of the concept they point at, and older rows without
to_source_idkeep their external URL and version. - #3013: only targets in the source being cloned from are moved to their equivalent.
- Repair command: it changes HEAD rows only, finds only the sources that need repair, and repairs each in its own transaction. The #3013 mappings are listed with both URLs and their external
to_source_urlis kept, again on a second run. Empty URLs are handled, including on sources with a blank canonical URL.
Before running the repair in production: run --dry-run and save the list of skipped mappings. It's the record for the #3013 hand fix, because the run then repairs those mappings' from_source_url.
Open (non-blocking): a missing blank line after the new test (inline).
Checks: pylint -j0 core/ exits 0 locally. 987 tests passed locally at c73da962, and the clone API tests (9) pass at 4285451b. The one failure, test_grandfathered_npm_url_is_downloaded_and_handed_over_as_upload, fails the same on master. CI on 4285451b is still pending (Pylint queued).
snyaggarwal
left a comment
There was a problem hiding this comment.
Review verdict: ready to merge (no findings)
Reviewed at 2643c6b1 (seventh pass). The only change since the last pass adds the missing blank line after the $clone API test (PEP 8 E302). I also re-read the whole diff, including the test-file changes, and found nothing new. Every earlier finding is fixed, and each thread has a reply.
Summary of the PR as it stands:
- #3012: copied mappings take the source ID, URL and (cleared) version of the concept they now point at. Older rows without
to_source_idkeep their external URL and pinned version. Equivalency mappings get their source URLs. - #3013: only targets in the source being cloned from are moved to their equivalent in the target. Mappings to other code systems are copied unchanged, and a self-mapping stays a self-mapping.
repair_mapping_source_urls:- It changes HEAD rows only, finds only the sources that need repair, and repairs each in its own transaction.
- Missing URLs (NULL or empty, including on sources with a blank canonical URL) and URLs naming another source are fixed.
- The #3013 mappings are listed with both URLs and their external
to_source_urlis kept, again on a second run.
- Tests: model-level clone tests, a repair-command test (dry run, a real run, and a second run), and an end-to-end
$cloneAPI test. The #3013 and blank-canonical cases fail without their fixes.
Before running the repair in production: run --dry-run and save the list of skipped mappings. It's the record for the #3013 hand fix, because the run then repairs those mappings' from_source_url.
Checks: pylint -j0 core/ exits 0 locally. 987 tests passed locally at c73da962, and the clone API tests passed at 4285451b. Since then only that test and this whitespace line changed. The one failure, test_grandfathered_npm_url_is_downloaded_and_handed_over_as_upload, fails the same on master. CI on 2643c6b1: Snyk passed, Pylint is still queued.
Fixes OpenConceptLab/ocl_issues#3012 and OpenConceptLab/ocl_issues#3013.
Why one PR: both bugs are in the same few lines of the
$clonepath (Source.clone_resources→Mapping.clone→Source.clone_mappings). #3013 decides which concept a copied mapping points at, and #3012 makes the source fields follow that concept. Fixed apart, the second PR would rewrite the same lines and conflict. The #3012 repair also has to know about #3013's mappings: on those,to_source_urlis the only record of the external code system they used to point at, and repairing #3012 alone would overwrite it.#3012: copied mappings keep the original's source URLs
Mapping.clone()copied the original'sfrom_source_url/to_source_urland source versions even when it swapped in a concept from the target source, andclone_mappings()updated only the source IDs. So a copied mapping'sfrom_source_urlstill named the source it was cloned from, and a mapping pointed at an equivalent concept kept the oldto_source_url. The equivalency mappings$clonecreates had no source URLs at all.Mapping.clone(user, from_concept, to_concept): when a substituted concept is in a different source from the original mapping's concept, the copy's source ID and URL are set to that concept's source, and the source version (which belonged to the old source) is cleared. The check compares the concepts' sources, not the stored source ID, so older rows with noto_source_idkeep their external URL (for example a canonical URL) and pinned version. The from and to sides share one helper,Mapping.get_source_fields_from_concept.Source.clone_mappings: fills a missing source ID or URL from the linked concept and keeps a recorded URL and version. The by-mnemonic re-resolution of unsaved concepts is removed. Its only caller,clone_resources, always passes saved concepts, so only tests used it.#3013: mappings to other code systems were retargeted
clone_resourcescalledget_equivalent_concept()for every copied mapping. If the target source had aSAME-ASto the mapping's target, the copy was pointed at that concept even when the target was LOINC or ICD-10, and the external code was lost.SAME-AS CIEL:136713on CIEL 136713) still becomes a self-mapping on the new concept. The original is self-referencing too, so that's the faithful copy, and with #3012 itsto_source_urlnow matches.select_related('parent'), the cascade mappings withselect_related('from_concept', 'to_concept'), and the equivalent concept with its source. So the source check,Mapping.cloneand the URL fill add no queries per mapping.Repair command:
repair_mapping_source_urlsAs asked in the #3012 comments. Use
--dry-runto list only.id = versioned_object_id). Mapping versions are not changed, so released source versions and their exports stay as they were. A mapping's versions list and version detail, and collection expansions that hold a version, also keep the old URL. The next edit of a mapping creates its new version from the repaired HEAD.to_conceptis in its own source butto_source_urlnames a source other than itsfrom_source_url(the source it was cloned from), its external target was replaced. The two URLs are compared by the source they name, URI or canonical URL. The command lists each such mapping with both URLs so it can be fixed by hand, and leaves itsto_source_urlin place. The same run then repairsfrom_source_url, so this output is the record of the cloned-from source. The to side runs before the from side so this comparison sees the originalfrom_source_url, and a later run still skips these mappings. Retargeted answers, whoseto_source_urlnames the cloned-from source, are repaired as usual. The command doesn't restore the external codes: the copy no longer has them, and #3013 plans a hand fix.Test changes
Two tests from #989 (
cloned_mapping_in_local_sourcein the importer tests andtest_put_200_keeps_the_local_link_of_a_cloned_mapping) relied on$cloneleaving the origin'sto_source_url. They now set that state explicitly after the clone, so the edit and import behavior for mappings cloned before this fix stays covered.test_clone_mappings_resolves_missing_concepts_by_mnemonicis removed along with the re-resolution.New tests:
from_source_urlwith no version, the equivalency mapping'sto_source_urlis the original source, and the external side keeps its URL and version.to_source_idand a canonicalto_source_url: the URL and pinned version are kept.to_source_urlis the target and its version is cleared.BROADER-THANmapping to an external concept that the target has aSAME-ASto: the copy still points at the external concept and source. This test fails without the fix.SAME-ASto itself: the copy is a self-mapping on the cloned concept, with the target's source URLs.POST .../concepts/$clone/withmapTypes=Q-AND-A,SAME-ASandequivalencyMapType=SAME-AS(a staff user, so no clone budget): the Q-AND-A answer moves to its existing equivalent with the target's source URLs, and theSAME-ASto an external concept keeps its concept, code and source, even though the target has aSAME-ASto that concept. This test fails without the #3013 fix.to_source_urlkept.to_source_url.pylint -j0 core/exits 0. Locally, the source, mapping, concept, bundle, importer, source API, mapping API and cross-repo access tests pass (987). The one failure,test_grandfathered_npm_url_is_downloaded_and_handed_over_as_upload, fails the same on master.🤖 Generated with Claude Code