Release 2.0.0-alpha.8

.github/workflows/javadoc.yml

@@ -0,0 +1,21 @@
+name: Check the javadoc
+
+on:
+  push:
+
+  # Allows to run this workflow manually from the Actions tab
+  workflow_dispatch:
+
+jobs:
+  javadoc:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-java@v4
+        with:
+          java-version: '11'
+          distribution: 'adopt'
+      - name: Check javadoc
+        run: mvn --batch-mode clean test-compile javadoc:javadoc javadoc:test-javadoc

.github/workflows/publish.yml

@@ -4,10 +4,9 @@ on:
   push:
     branches:
       - 'develop'
-      - 'main'
 
   release:
-    types: [created]
+    types: [published]
 
   # Allows to run this workflow manually from the Actions tab
   workflow_dispatch:

CHANGELOG.md

@@ -1,19 +1,21 @@
-# EFX Toolkit 2.0.0-alpha.7 Release Notes
+# EFX Toolkit 2.0.0-alpha.8 Release Notes
 
 _The EFX Toolkit for Java developers is a library that enables the transpilation of [EFX](https://docs.ted.europa.eu/eforms/latest/efx) expressions and templates to different target languages. It also includes an implementation of an EFX-to-XPath transpiler._
 
 ---
 
 ## In this release
 
-This is an incremental update over 2.0.0-alpha.6. It refines SDK data-type resolution and updates the eForms Core dependency, while retaining full support for the EFX-2 grammar released with SDK 2.0.0-alpha.2.
+This is an incremental update over 2.0.0-alpha.7.
 
-### Changes since 2.0.0-alpha.6
+### Changes since 2.0.0-alpha.7
 
-- Field data types are now taken directly from the SDK field definitions, removing a temporary `SdkSymbolResolver` heuristic that inferred the `duration` type from `measure` fields using the `duration-unit` codelist. Added regression tests covering measure and duration fields.
-- Upgraded the eForms Core Java dependency to 1.7.0.
+- **Selectors**: an EFX expression can now yield the path of the XML elements a reference points to, instead of the values held in them. Write `&{reference}` in EFX-1, and `&{reference}` or `WITH context SELECT reference` in EFX-2. The first use of this new feature is in the `privacy.undisclosedFieldSelector` property in `fields.json` of SDK 1.16.0-beta.2 and SDK 2.0.0-alpha.3. The new property identifies the elements that must be masked and effectively withheld from publication.
+- **Preferred language selection**: fixed an issue with the implicit invocation in EFX-1 of the `preferred-language-text` function when multilingual text fields are referenced. The function is now called correctly only when transpiling view templates, which is the only context where this functionality is applicable. EFX-2 requires explicit invocation of the function and is therefore unaffected by this fix.
+- **Context overrides**: fixed multiple issues that caused the transpiler to produce valid but inaccurate XPaths when using a context override (`context::field`) to modify the path through which the value of a field is reached. Transpilation now preserves navigation steps and predicates accurately, and expressions using a context override will now produce a longer XPath, which however honours all predicates and enforces the presence of the context to select the designated value(s). This feature has not been used so far in any published SDK, so this change does not have an impact on the interpretation of existing rules and templates published in any SDK version.
+- The eForms Core Java dependency is now 1.9.0.
 
-The following sections describe the features of the 2.0.0 line, unchanged since 2.0.0-alpha.6.
+The following sections describe the features of the 2.0.0 line, unchanged since 2.0.0-alpha.7.
 
 ### EFX-2 language support
 
@@ -59,7 +61,7 @@ NOTE: Transpilation of EFX-1 to XPath and XSL in this version of the EFX Toolkit
 
 ## Breaking changes
 
-No new breaking changes are introduced in 2.0.0-alpha.7. For the breaking changes introduced earlier in the 2.0.0 line, see the [2.0.0-alpha.6 release notes](https://github.com/OP-TED/efx-toolkit-java/releases/tag/2.0.0-alpha.6).
+`XPathContextualizer.join` now takes a third argument, which says how far the joined path may be shortened; the two-argument form has been removed. For the breaking changes introduced earlier in the 2.0.0 line, see the [2.0.0-alpha.6 release notes](https://github.com/OP-TED/efx-toolkit-java/releases/tag/2.0.0-alpha.6).
 
 ## Future development
 
@@ -76,7 +78,7 @@ Documentation for the EFX Toolkit is available at: <https://docs.ted.europa.eu/e
 
 This version of the EFX Toolkit has a compile-time dependency on the following eForms SDK versions and uses the EFX grammar that each version provides:
 
-- eForms SDK 1.x.x (SDK 1.13.0 grammar)
-- eForms SDK 2.0.0-alpha.2
+- eForms SDK 1.16.0-beta.2
+- eForms SDK 2.0.0-alpha.3
 
-It also depends on the [eForms Core Java library](https://github.com/OP-TED/eforms-core-java) version 1.7.0.
+It also depends on the [eForms Core Java library](https://github.com/OP-TED/eforms-core-java) version 1.9.0.

pom.xml

@@ -3,7 +3,7 @@
 
   <groupId>eu.europa.ted.eforms</groupId>
   <artifactId>efx-toolkit-java</artifactId>
-  <version>2.0.0-alpha.7</version>
+  <version>2.0.0-alpha.8</version>
   <packaging>jar</packaging>
 
   <name>EFX Toolkit for Java</name>
@@ -46,9 +46,9 @@
     <sdk.antlr4.dir>${project.build.directory}/eforms-sdk/antlr4</sdk.antlr4.dir>
 
     <!-- Versions - eForms -->
-    <version.eforms-sdk-1>1.13.0</version.eforms-sdk-1>
-    <version.eforms-sdk-2>2.0.0-alpha.2</version.eforms-sdk-2>
-    <version.eforms-core>1.7.0</version.eforms-core>
+    <version.eforms-sdk-1>1.16.0-beta.2</version.eforms-sdk-1>
+    <version.eforms-sdk-2>2.0.0-alpha.3</version.eforms-sdk-2>
+    <version.eforms-core>1.9.0</version.eforms-core>
 
     <!-- Versions - Third-party libraries -->
     <version.antlr4>4.13.1</version.antlr4>

src/main/java/eu/europa/ted/efx/interfaces/ScriptGenerator.java

@@ -328,7 +328,12 @@ default PathExpression composeFieldInExternalReference(final PathExpression exte
   /**
    * Joins two given path expressions into one by placing the second after the first and using the
    * proper delimiter.
-   * 
+   *
+   * <p>
+   * This is how the path of a context override is built: the second path is relative to the first,
+   * which is the context the override names. What that implies for the joined path is for the
+   * implementation to decide, in the terms of the language it targets.
+   *
    * @param first The part of the path that goes before the delimiter.
    * @param second The part of the path that goes after the delimiter.
    * @return The joined path expression.
@@ -799,7 +804,9 @@ public StringExpression composeReplaceRegexFunction(StringExpression text,
    * Given a reference to a multilingual field, this function should generate the target language script
    * that returns the text value of the field in the preferred language.
    * 
-   * Calling the function in EFX 2
+   * In EFX-2 the template author calls this explicitly, through the
+   * {@code preferred-language-text} function. EFX-1 has no such function, so template
+   * translation applies it implicitly to every multilingual field it renders.
    * 
    * @since SDK 2.0.0
    * @see #getPreferredLanguage(PathExpression)

src/main/java/eu/europa/ted/efx/sdk1/EfxExpressionTranslatorV1.java

@@ -1095,6 +1095,23 @@ public void exitSequenceFromFieldReference(SequenceFromFieldReferenceContext ctx
     }
   }
 
+  /**
+   * A selector-block yields the reference itself rather than its value: the value step that every
+   * other reference position applies is deliberately not composed here. The path is otherwise
+   * resolved exactly as it would be in an expression-block, relative to the declared context
+   * unless the author wrote it as an absolute reference.
+   */
+  @Override
+  public void exitSelection(final SelectionContext ctx) {
+    if (ctx.attributeReference() != null) {
+      // attributeReference has no exit handler of its own, so the attribute step is composed here
+      // rather than globally, which would double-compose it for the scalar and sequence positions.
+      this.stack.push(this.script.composeFieldAttributeReference(
+          this.stack.pop(PathExpression.class),
+          ctx.attributeReference().Identifier().getText(), StringPath.class));
+    }
+  }
+
   @Override
   public void exitScalarFromAttributeReference(ScalarFromAttributeReferenceContext ctx) {
     this.stack.push(this.script.composeFieldAttributeReference(this.stack.pop(PathExpression.class),
@@ -1117,12 +1134,11 @@ public void exitSequenceFromAttributeReference(SequenceFromAttributeReferenceCon
    */
   @Override
   public void exitContextFieldSpecifier(ContextFieldSpecifierContext ctx) {
-    this.stack.pop(PathExpression.class); // Discard the PathExpression placed in the stack for
-                                          // the context field.
+    final PathExpression contextFieldPath = this.stack.pop(PathExpression.class);
     final String contextFieldId = getFieldId(ctx.fieldContext());
     this.efxContext
         .push(new FieldContext(contextFieldId, this.symbols.getAbsolutePathOfField(contextFieldId),
-            this.symbols.getRelativePathOfField(contextFieldId, this.efxContext.symbol())));
+            contextFieldPath));
   }
 
 
@@ -1146,12 +1162,11 @@ public void exitFieldReferenceWithFieldContextOverride(
    */
   @Override
   public void exitContextNodeSpecifier(ContextNodeSpecifierContext ctx) {
-    this.stack.pop(PathExpression.class); // Discard the PathExpression placed in the stack for
-                                          // the context node.
+    final PathExpression contextNodePath = this.stack.pop(PathExpression.class);
     final String contextNodeId = getNodeId(ctx.node);
     this.efxContext
         .push(new NodeContext(contextNodeId, this.symbols.getAbsolutePathOfNode(contextNodeId),
-            this.symbols.getRelativePathOfNode(contextNodeId, this.efxContext.symbol())));
+            contextNodePath));
   }
 
   /**

src/main/java/eu/europa/ted/efx/sdk1/EfxTemplateTranslatorV1.java

@@ -359,7 +359,7 @@ private void shorthandIndirectLabelReference(ParserRuleContext ctx, final String
         ? this.script.composeFieldAttributeReference(
             this.script.contextualizePath(this.symbols.getAbsolutePathOfFieldWithoutTheAttribute(fieldId), currentContext.absolutePath()),
             this.symbols.getAttributeNameFromAttributeField(fieldId), StringPath.class)
-        : this.script.composeFieldValueReference(
+        : this.composeFieldValueReference(
         this.symbols.getRelativePathOfField(fieldId, currentContext.symbol()));
     Variable loopVariable = new Variable("item",
         this.script.composeVariableDeclaration("item", StringExpression.class), StringExpression.empty(),
@@ -532,12 +532,60 @@ public void exitShorthandFieldValueReferenceFromContextField(
     if (!this.efxContext.isFieldContext()) {
       throw InvalidUsageException.shorthandRequiresFieldContext(ctx, "$value");
     }
-    this.stack.push(this.script.composeFieldValueReference(
+    this.stack.push(this.composeFieldValueReference(
         this.symbols.getRelativePathOfField(this.efxContext.symbol(), this.efxContext.symbol())));
   }
 
   // #endregion Expression Blocks ${...} --------------------------------------
-
+
+  // #region Value References -------------------------------------------------
+
+  /***
+   * Multilingual fields are handled by this class, so the value reference is composed here instead
+   * of directly by the script generator. Anything else is left to the inherited behaviour.
+   *
+   * @see #composeFieldValueReference(PathExpression)
+   */
+  @Override
+  public void exitScalarFromFieldReference(final ScalarFromFieldReferenceContext ctx) {
+    if (!this.stack.peekType().is(EfxDataType.MultilingualString.class)) {
+      super.exitScalarFromFieldReference(ctx);
+      return;
+    }
+    this.stack.push(this.composeFieldValueReference(this.stack.pop(PathExpression.class)));
+  }
+
+  /***
+   * @see #exitScalarFromFieldReference(ScalarFromFieldReferenceContext)
+   */
+  @Override
+  public void exitSequenceFromFieldReference(final SequenceFromFieldReferenceContext ctx) {
+    if (!this.stack.peekType().is(EfxDataType.MultilingualString.class)) {
+      super.exitSequenceFromFieldReference(ctx);
+      return;
+    }
+    this.stack.push(this.composeFieldValueReference(this.stack.pop(PathExpression.class)));
+  }
+
+  /***
+   * In a view template the value of a multilingual field must be rendered in the language preferred
+   * by the reader, which EFX-1 gives the template author no syntax to ask for. Template translation
+   * therefore selects the preferred language implicitly, for every multilingual field it renders.
+   *
+   * Outside of view templates no such selection is possible: the function that performs it is
+   * provided by the XSL of the notice viewer and exists nowhere else. There the value of a
+   * multilingual field is retrieved like that of any other text field.
+   */
+  private PathExpression composeFieldValueReference(final PathExpression fieldReference) {
+    if (fieldReference.is(EfxDataType.MultilingualString.class)) {
+      return Expression.from(this.script.getTextInPreferredLanguage(fieldReference),
+          fieldReference.getClass());
+    }
+    return this.script.composeFieldValueReference(fieldReference);
+  }
+
+  // #endregion Value References ----------------------------------------------
+
   // #region Context Declaration Blocks {...} ---------------------------------
 
   /**

src/main/java/eu/europa/ted/efx/sdk1/xpath/XPathScriptGeneratorV1.java

@@ -25,7 +25,6 @@
 import eu.europa.ted.efx.model.expressions.scalar.NumericExpression;
 import eu.europa.ted.efx.model.expressions.scalar.StringExpression;
 import eu.europa.ted.efx.model.expressions.scalar.StringLiteral;
-import eu.europa.ted.efx.model.types.EfxDataType;
 import eu.europa.ted.efx.xpath.XPathScriptGenerator;
 
 @SdkComponent(versions = {"1"}, componentType = SdkComponentType.SCRIPT_GENERATOR)
@@ -48,29 +47,6 @@ public StringExpression composeToStringConversion(NumericExpression number) {
         return new StringExpression("format-number(" + number.getScript() + ", '" + formatString + "')");
     }
 
-    /***
-     * This method is overridden to workaround a limitation of EFX 1.
-     * 
-     * When a multilingual text field is referenced, then a special XPath expression
-     * is generated to retrieve the value in the "preferred" language.
-     * Preferred language is the first language among the languages listed in the
-     * translator options for which a text value is available in the field.
-     * 
-     * The logic of the workaround is as follows:
-     * if the fieldReference is a multilingual text field and it does not
-     * already come with a predicate that filters by @languageID, then we add a
-     * predicate which, using a for loop, will find the first language for which a
-     * value is available in the field.
-     * 
-     * In EFX 1 therefore the selection of the appropriate (preferred) language is
-     * done implicitly, whereas in EFX 2 it is done explicitly by calling a special
-     * function designed to perform this task.
-     * 
-     * Both EFX-1 and EFX-2 implementations of the feature rely on the existence of a
-     * $PREFERRED_LANGUAGES variable in the XSLT.
-     * This function returns the list of languages used in the visualisation in the
-     * order of preference (visualisation language followed by notice language(s)).
-     */
     /**
      * Preserved V1 behavior: pass EFX string literal through as-is without converting
      * escape sequences to XPath format.
@@ -91,13 +67,31 @@ public BooleanExpression composePatternMatchCondition(StringExpression expressio
             String.format("fn:matches(normalize-space(%s), %s)", expression.getScript(), pattern));
     }
 
+    /***
+     * Retrieves the value of a multilingual text field in the "preferred" language.
+     * Preferred language is the first language among the languages listed in the
+     * translator options for which a text value is available in the field.
+     *
+     * This is a workaround for a limitation of EFX 1: the language cannot be selected
+     * explicitly by the template author, so template translation applies this
+     * implicitly to every multilingual field it renders. In EFX 2 the selection is
+     * done explicitly, by calling a function designed to perform this task.
+     *
+     * If the reference already comes with a predicate that filters by @languageID,
+     * then the template author has already pinned a language and the value is
+     * retrieved as-is.
+     */
     @Override
-    public PathExpression composeFieldValueReference(PathExpression fieldReference) {
-        XPathInfo xpathInfo = XPathProcessor.parse(fieldReference.getScript());
-        if (fieldReference.is(EfxDataType.MultilingualString.class) && !xpathInfo.hasPredicate("@languageID")) {
-            return Expression.instantiate("efx:preferred-language-text(" + fieldReference.getScript() + ")", fieldReference.getClass());
+    public StringExpression getTextInPreferredLanguage(final PathExpression fieldReference) {
+        final XPathInfo xpathInfo = XPathProcessor.parse(fieldReference.getScript());
+        if (xpathInfo.hasPredicate("@languageID")) {
+            // The value reference is a PathExpression, which is not a StringExpression and cannot
+            // be returned as such. Only the generated script matters here: the caller re-creates
+            // the expression using the type of the field reference it started from.
+            return Expression.from(super.composeFieldValueReference(fieldReference),
+                StringExpression.class);
         }
-        return super.composeFieldValueReference(fieldReference);
+        return super.getTextInPreferredLanguage(fieldReference);
     }
 
     @Override

src/main/java/eu/europa/ted/efx/sdk2/EfxExpressionTranslatorV2.java

@@ -1919,6 +1919,25 @@ public void exitSequenceFromFieldReference(SequenceFromFieldReferenceContext ctx
     this.resolveAndPushFieldReference(ctx, result, fieldId);
   }
 
+  /**
+   * A selector yields the reference itself rather than its value: the value step that every other
+   * reference position applies is deliberately not composed here. The path is otherwise resolved
+   * exactly as it would be in an expression, relative to the declared context unless the author
+   * wrote it as an absolute reference.
+   *
+   * <p>The reference tier has already left the path on the stack. Only an attribute reference
+   * needs work, because {@code attributeReference} has no exit handler of its own; composing the
+   * attribute step globally would double-compose it for the scalar and sequence positions.
+   */
+  @Override
+  public void exitSelection(final SelectionContext ctx) {
+    if (ctx.attributeReference() != null) {
+      this.stack.push(this.script.composeFieldAttributeReference(
+          this.stack.pop(PathExpression.class),
+          ctx.attributeReference().attributeName.getText(), StringPath.class));
+    }
+  }
+
   @Override
   public void exitScalarFromAttributeReference(ScalarFromAttributeReferenceContext ctx) {
     PathExpression result = this.script.composeFieldAttributeReference(this.stack.pop(PathExpression.class),
@@ -2055,12 +2074,11 @@ public void exitFieldReferenceWithFieldContextOverride(
    */
   @Override
   public void exitContextNodeSpecifier(ContextNodeSpecifierContext ctx) {
-    this.stack.pop(PathExpression.class); // Discard the PathExpression placed in the stack for
-                                          // the context node.
+    final PathExpression contextNodePath = this.stack.pop(PathExpression.class);
     final String contextNodeId = getNodeId(ctx.node);
     this.efxContext
         .push(new NodeContext(contextNodeId, this.symbols.getAbsolutePathOfNode(contextNodeId),
-            this.symbols.getRelativePathOfNode(contextNodeId, this.efxContext.symbol())));
+            contextNodePath));
   }
 
   /**