Make the WARC digest algorithm configurable

[abhinav-phi]

This PR addresses #2106 and fixes it.

Problem

WARCRecordFormat.getDigestSha1() computed a SHA-1 digest over the bytes and returned it as sha1:<base32>. That fixed value was used for the WARC-Payload-Digest and WARC-Block-Digest headers of response, resource, request and metadata records. There was no configuration key for the algorithm, so an operator who wants SHA-256 digests in their archives — which WARC 1.1 allows — could not have them without rewriting the records afterwards.

SHA-1 digests are the convention across the WARC ecosystem and identify content for deduplication rather than authenticate it, so this is not urgent — but it was still a fixed choice in a place where the digested bytes come from the network. Operators whose own policy rules out SHA-1, or whose downstream tooling wants SHA-256, had no option.

Solution

Add a configuration key warc.digest.algorithm and thread the configured algorithm through all record formats:

warc.digest.algorithm: sha256

• Accepted values: sha1 and sha256. The value is matched case-insensitively, an optional hyphen is ignored (sha256, SHA-256, SHA256 are all accepted) and surrounding whitespace is trimmed.
• sha1 remains the default: CDX indexes and revisit tooling downstream assume sha1: base32, and changing the default would break them. Deciding on a different default can be done separately from adding the option.
• Fail fast on invalid values: an unsupported value (e.g. md5) raises an IllegalArgumentException with a descriptive message when the bolt is prepared, instead of silently producing digests with a different algorithm than the one configured.

Changes

external/warc/src/main/java/org/apache/stormcrawler/warc/WARCRecordFormat.java

• New configuration key constants: DIGEST_ALGORITHM_PARAM (warc.digest.algorithm) and the supported values DIGEST_ALGORITHM_SHA1 / DIGEST_ALGORITHM_SHA256.
• New constructor WARCRecordFormat(String protocolMDprefix, String digestAlgorithm); the existing single-argument constructor is unchanged in behaviour and keeps defaulting to SHA-1.
• New instance methods getDigest(byte[]) and getDigest(byte[], byte[]) emit the matching sha1: / sha256: prefix. The digest used for records without content (digestNoContent) is now derived from the configured algorithm instead of a static SHA-1 constant.
• The static helpers getDigestSha1(byte[]) and getDigestSha1(byte[], byte[]) are kept for compatibility with code outside the module that calls them, now marked @Deprecated in favour of the instance methods.

external/warc/src/main/java/org/apache/stormcrawler/warc/WARCRequestRecordFormat.java and MetadataRecordFormat.java

• New constructors accepting the digest algorithm; both use the configurable getDigest instance methods so request and metadata records use the same algorithm as response and resource records.

external/warc/src/main/java/org/apache/stormcrawler/warc/WARCHdfsBolt.java

• Reads warc.digest.algorithm from the topology configuration (default sha1) and passes it to every record format it instantiates, so a single setting governs the whole WARC output.

Documentation

• external/warc/README.md: documents the new key, the accepted values and the compatibility considerations around SHA-1.
• docs/src/main/asciidoc/configuration.adoc: adds warc.digest.algorithm to the WARC section of the configuration reference.

Note on #2034

As requested in the issue, I checked PR #2034 ("WARC writer: WARC-Protocol header to follow WARC field proposals"): it is about the WARC-Protocol / cipher-suite headers and does not touch digest computation, so this is not a duplicate. It modifies the same file, so whichever merges second may need a trivial rebase.

Testing

• New WARCDigestAlgorithmTest (9 tests):
  • SHA-1 is the default, including for a null algorithm value;
  • SHA-256 digests for getDigest(byte[]) and getDigest(byte[], byte[]), including empty content;
  • acceptance of value variants (SHA256, SHA-256, sha256, SHA-1);
  • IllegalArgumentException for unsupported values (md5, sha512) on all three record format classes;
  • full response, resource, request and metadata records verifying that WARC-Payload-Digest / WARC-Block-Digest carry the sha256: prefix and the correct digest value, computed independently of the code under test.
• WARCHdfsBoltTest.testDigestAlgorithmConfig prepares the bolt with warc.digest.algorithm: sha256 and verifies that the written records (warcinfo, request, response) use sha256: digests.
• mvn -pl external/warc test: 22 tests run, 0 failures, 0 errors; checkstyle:check reports 0 violations.