Sync develop after 2.0.0-alpha.8 and set the next snapshot

CHANGELOG.md

@@ -1,19 +1,21 @@
-# EFX Toolkit 2.0.0-alpha.7 Release Notes
+# EFX Toolkit 2.0.0-alpha.8 Release Notes
 
 _The EFX Toolkit for Java developers is a library that enables the transpilation of [EFX](https://docs.ted.europa.eu/eforms/latest/efx) expressions and templates to different target languages. It also includes an implementation of an EFX-to-XPath transpiler._
 
 ---
 
 ## In this release
 
-This is an incremental update over 2.0.0-alpha.6. It refines SDK data-type resolution and updates the eForms Core dependency, while retaining full support for the EFX-2 grammar released with SDK 2.0.0-alpha.2.
+This is an incremental update over 2.0.0-alpha.7.
 
-### Changes since 2.0.0-alpha.6
+### Changes since 2.0.0-alpha.7
 
-- Field data types are now taken directly from the SDK field definitions, removing a temporary `SdkSymbolResolver` heuristic that inferred the `duration` type from `measure` fields using the `duration-unit` codelist. Added regression tests covering measure and duration fields.
-- Upgraded the eForms Core Java dependency to 1.7.0.
+- **Selectors**: an EFX expression can now yield the path of the XML elements a reference points to, instead of the values held in them. Write `&{reference}` in EFX-1, and `&{reference}` or `WITH context SELECT reference` in EFX-2. The first use of this new feature is in the `privacy.undisclosedFieldSelector` property in `fields.json` of SDK 1.16.0-beta.2 and SDK 2.0.0-alpha.3. The new property identifies the elements that must be masked and effectively withheld from publication.
+- **Preferred language selection**: fixed an issue with the implicit invocation in EFX-1 of the `preferred-language-text` function when multilingual text fields are referenced. The function is now called correctly only when transpiling view templates, which is the only context where this functionality is applicable. EFX-2 requires explicit invocation of the function and is therefore unaffected by this fix.
+- **Context overrides**: fixed multiple issues that caused the transpiler to produce valid but inaccurate XPaths when using a context override (`context::field`) to modify the path through which the value of a field is reached. Transpilation now preserves navigation steps and predicates accurately, and expressions using a context override will now produce a longer XPath, which however honours all predicates and enforces the presence of the context to select the designated value(s). This feature has not been used so far in any published SDK, so this change does not have an impact on the interpretation of existing rules and templates published in any SDK version.
+- The eForms Core Java dependency is now 1.9.0.
 
-The following sections describe the features of the 2.0.0 line, unchanged since 2.0.0-alpha.6.
+The following sections describe the features of the 2.0.0 line, unchanged since 2.0.0-alpha.7.
 
 ### EFX-2 language support
 
@@ -59,7 +61,7 @@ NOTE: Transpilation of EFX-1 to XPath and XSL in this version of the EFX Toolkit
 
 ## Breaking changes
 
-No new breaking changes are introduced in 2.0.0-alpha.7. For the breaking changes introduced earlier in the 2.0.0 line, see the [2.0.0-alpha.6 release notes](https://github.com/OP-TED/efx-toolkit-java/releases/tag/2.0.0-alpha.6).
+`XPathContextualizer.join` now takes a third argument, which says how far the joined path may be shortened; the two-argument form has been removed. For the breaking changes introduced earlier in the 2.0.0 line, see the [2.0.0-alpha.6 release notes](https://github.com/OP-TED/efx-toolkit-java/releases/tag/2.0.0-alpha.6).
 
 ## Future development
 
@@ -76,7 +78,7 @@ Documentation for the EFX Toolkit is available at: <https://docs.ted.europa.eu/e
 
 This version of the EFX Toolkit has a compile-time dependency on the following eForms SDK versions and uses the EFX grammar that each version provides:
 
-- eForms SDK 1.x.x (SDK 1.13.0 grammar)
-- eForms SDK 2.0.0-alpha.2
+- eForms SDK 1.16.0-beta.2
+- eForms SDK 2.0.0-alpha.3
 
-It also depends on the [eForms Core Java library](https://github.com/OP-TED/eforms-core-java) version 1.7.0.
+It also depends on the [eForms Core Java library](https://github.com/OP-TED/eforms-core-java) version 1.9.0.

pom.xml

@@ -48,7 +48,7 @@
     <!-- Versions - eForms -->
     <version.eforms-sdk-1>1.16.0-SNAPSHOT</version.eforms-sdk-1>
     <version.eforms-sdk-2>2.0.0-SNAPSHOT</version.eforms-sdk-2>
-    <version.eforms-core>1.8.0-SNAPSHOT</version.eforms-core>
+    <version.eforms-core>1.10.0-SNAPSHOT</version.eforms-core>
 
     <!-- Versions - Third-party libraries -->
     <version.antlr4>4.13.1</version.antlr4>

src/main/java/eu/europa/ted/efx/interfaces/ScriptGenerator.java

@@ -328,7 +328,12 @@ default PathExpression composeFieldInExternalReference(final PathExpression exte
   /**
    * Joins two given path expressions into one by placing the second after the first and using the
    * proper delimiter.
-   * 
+   *
+   * <p>
+   * This is how the path of a context override is built: the second path is relative to the first,
+   * which is the context the override names. What that implies for the joined path is for the
+   * implementation to decide, in the terms of the language it targets.
+   *
    * @param first The part of the path that goes before the delimiter.
    * @param second The part of the path that goes after the delimiter.
    * @return The joined path expression.
@@ -799,7 +804,9 @@ public StringExpression composeReplaceRegexFunction(StringExpression text,
    * Given a reference to a multilingual field, this function should generate the target language script
    * that returns the text value of the field in the preferred language.
    * 
-   * Calling the function in EFX 2
+   * In EFX-2 the template author calls this explicitly, through the
+   * {@code preferred-language-text} function. EFX-1 has no such function, so template
+   * translation applies it implicitly to every multilingual field it renders.
    * 
    * @since SDK 2.0.0
    * @see #getPreferredLanguage(PathExpression)

src/main/java/eu/europa/ted/efx/sdk1/EfxExpressionTranslatorV1.java

@@ -1134,12 +1134,11 @@ public void exitSequenceFromAttributeReference(SequenceFromAttributeReferenceCon
    */
   @Override
   public void exitContextFieldSpecifier(ContextFieldSpecifierContext ctx) {
-    this.stack.pop(PathExpression.class); // Discard the PathExpression placed in the stack for
-                                          // the context field.
+    final PathExpression contextFieldPath = this.stack.pop(PathExpression.class);
     final String contextFieldId = getFieldId(ctx.fieldContext());
     this.efxContext
         .push(new FieldContext(contextFieldId, this.symbols.getAbsolutePathOfField(contextFieldId),
-            this.symbols.getRelativePathOfField(contextFieldId, this.efxContext.symbol())));
+            contextFieldPath));
   }
 
 
@@ -1163,12 +1162,11 @@ public void exitFieldReferenceWithFieldContextOverride(
    */
   @Override
   public void exitContextNodeSpecifier(ContextNodeSpecifierContext ctx) {
-    this.stack.pop(PathExpression.class); // Discard the PathExpression placed in the stack for
-                                          // the context node.
+    final PathExpression contextNodePath = this.stack.pop(PathExpression.class);
     final String contextNodeId = getNodeId(ctx.node);
     this.efxContext
         .push(new NodeContext(contextNodeId, this.symbols.getAbsolutePathOfNode(contextNodeId),
-            this.symbols.getRelativePathOfNode(contextNodeId, this.efxContext.symbol())));
+            contextNodePath));
   }
 
   /**

src/main/java/eu/europa/ted/efx/sdk1/xpath/XPathScriptGeneratorV1.java

@@ -80,11 +80,6 @@ public BooleanExpression composePatternMatchCondition(StringExpression expressio
      * If the reference already comes with a predicate that filters by @languageID,
      * then the template author has already pinned a language and the value is
      * retrieved as-is.
-     *
-     * Both EFX-1 and EFX-2 implementations of the feature rely on the existence of a
-     * $PREFERRED_LANGUAGES variable in the XSLT.
-     * This function returns the list of languages used in the visualisation in the
-     * order of preference (visualisation language followed by notice language(s)).
      */
     @Override
     public StringExpression getTextInPreferredLanguage(final PathExpression fieldReference) {

src/main/java/eu/europa/ted/efx/sdk2/EfxExpressionTranslatorV2.java

@@ -2074,12 +2074,11 @@ public void exitFieldReferenceWithFieldContextOverride(
    */
   @Override
   public void exitContextNodeSpecifier(ContextNodeSpecifierContext ctx) {
-    this.stack.pop(PathExpression.class); // Discard the PathExpression placed in the stack for
-                                          // the context node.
+    final PathExpression contextNodePath = this.stack.pop(PathExpression.class);
     final String contextNodeId = getNodeId(ctx.node);
     this.efxContext
         .push(new NodeContext(contextNodeId, this.symbols.getAbsolutePathOfNode(contextNodeId),
-            this.symbols.getRelativePathOfNode(contextNodeId, this.efxContext.symbol())));
+            contextNodePath));
   }
 
   /**

src/main/java/eu/europa/ted/efx/xpath/XPathContextualizer.java

@@ -13,6 +13,7 @@
  */
 package eu.europa.ted.efx.xpath;
 
+import eu.europa.ted.eforms.xpath.XPathProcessor.Simplification;
 import eu.europa.ted.eforms.xpath.XPathProcessor;
 import eu.europa.ted.efx.model.expressions.Expression;
 import eu.europa.ted.efx.model.expressions.PathExpression;
@@ -39,9 +40,15 @@ public static PathExpression contextualize(final PathExpression contextXpath,
     return Expression.instantiate(result, xpath.getClass());
   }
 
-  public static PathExpression join(final PathExpression first, final PathExpression second) {
+  /**
+   * Joins the path of a context to a path that is relative to it, shortening the result as far as
+   * the caller asks for.
+   */
+  public static PathExpression join(final PathExpression first, final PathExpression second,
+      final Simplification simplification) {
 
-    String joinedXPath = XPathProcessor.join(first.getScript(), second.getScript());
+    String joinedXPath =
+        XPathProcessor.join(first.getScript(), second.getScript(), simplification);
 
     return Expression.instantiate(joinedXPath, second.getClass());
   }

src/main/java/eu/europa/ted/efx/xpath/XPathScriptGenerator.java

@@ -27,6 +27,7 @@
 
 import eu.europa.ted.eforms.sdk.component.SdkComponent;
 import eu.europa.ted.eforms.sdk.component.SdkComponentType;
+import eu.europa.ted.eforms.xpath.XPathProcessor.Simplification;
 import eu.europa.ted.efx.interfaces.ScriptGenerator;
 import eu.europa.ted.efx.interfaces.TranslatorOptions;
 import eu.europa.ted.efx.model.expressions.Expression;
@@ -298,7 +299,12 @@ public <T extends Expression> T composeParenthesizedExpression(T expression, Cla
 
   @Override
   public PathExpression joinPaths(final PathExpression first, final PathExpression second) {
-    return XPathContextualizer.join(first, second);
+    // Every step of both paths is kept. A context override means the value of the field by a path
+    // that starts at the context, so where the context is not present in the notice there is no
+    // starting point and nothing is selected. Shortening the path would remove the walk to the
+    // context and with it that condition, leaving an override indistinguishable from a plain
+    // reference.
+    return XPathContextualizer.join(first, second, Simplification.NONE);
   }
 
   @Override
@@ -784,11 +790,27 @@ public StringLiteral getStringLiteralFromUnquotedString(String value) {
     return new StringLiteral("'" + value + "'");
   }
 
+  /**
+   * Emits a call to {@code efx:preferred-language}, a function of the notice viewer's XSLT runtime
+   * library. It returns the identifier of the first language, among those the visualisation
+   * prefers, for which the field holds a value.
+   *
+   * @see #getTextInPreferredLanguage(PathExpression)
+   */
   @Override
   public StringExpression getPreferredLanguage(PathExpression fieldReference) {
     return new StringExpression("efx:preferred-language(" + fieldReference.getScript() + ")");
   }
 
+  /**
+   * Emits a call to {@code efx:preferred-language-text}, a function of the notice viewer's XSLT
+   * runtime library.
+   *
+   * Both EFX-1 and EFX-2 reach this method, and the function they call relies on a
+   * {@code $PREFERRED_LANGUAGES} variable defined by the XSLT. That variable holds the languages
+   * used in the visualisation, in order of preference: the visualisation language followed by the
+   * notice languages.
+   */
   @Override
   public StringExpression getTextInPreferredLanguage(PathExpression fieldReference) {
     return new StringExpression("efx:preferred-language-text(" + fieldReference.getScript() + ")");

src/test/java/eu/europa/ted/efx/sdk1/EfxExpressionTranslatorV1Test.java

@@ -1089,13 +1089,21 @@ void testFieldReferenceInOtherNotice() {
 
   @Test
   void testFieldReferenceWithFieldContextOverride() {
-    testExpressionTranslationWithContext("../TextField/normalize-space(text())", "BT-00-Code",
+    testExpressionTranslationWithContext("../ChildNode/SubLevelTextField/../../TextField/normalize-space(text())", "BT-00-Code",
         "BT-01-SubLevel-Text::BT-00-Text");
   }
 
+  @Test
+  void testFieldReferenceWithFieldContextOverride_WithPredicate() {
+    testExpressionTranslationWithContext(
+        "../ChildNode/SubLevelTextField['a' = 'a']/../../TextField/normalize-space(text())",
+        "BT-00-Code",
+        "BT-01-SubLevel-Text['a' == 'a']::BT-00-Text");
+  }
+
   @Test
   void testFieldReferenceWithFieldContextOverride_WithIntegerField() {
-    testExpressionTranslationWithContext("../IntegerField/number()", "BT-00-Code",
+    testExpressionTranslationWithContext("../ChildNode/SubLevelTextField/../../IntegerField/number()", "BT-00-Code",
         "BT-01-SubLevel-Text::BT-00-Integer");
   }
 
@@ -1107,7 +1115,7 @@ void testFieldReferenceWithNodeContextOverride() {
 
   @Test
   void testFieldReferenceWithNodeContextOverride_WithPredicate() {
-    testExpressionTranslationWithContext("../../PathNode/IntegerField/number()", "BT-00-Text",
+    testExpressionTranslationWithContext("../..[PathNode/IndicatorField = true()]/PathNode/IntegerField/number()", "BT-00-Text",
         "ND-Root[BT-00-Indicator == TRUE]::BT-00-Integer");
   }