Add Cisco NBAR application reports with reusable history summaries

CHANGELOG.md

@@ -2,35 +2,35 @@
 
 --- develop ---
 
-* fix: Make bundled Net_DNS2 constructor nullability explicit for PHP 8.4 without changing accepted arguments
-
-* issue: Shorten the graph drilldown button title from 'View NetFlow Traffic In Range' to 'NetFlow' so it reads cleanly as a label in Cacti's graph action menu
-* dev: Create the plugin's config.php from config.php.dist for the flowview install step in the integration-test workflow (removed immediately afterwards so the unit tests still run with no config file), so the install no longer fails with "needs configuration"
-* feature: Add an optional FlowView Query Service (a long-running Go service under service/flowview-query) as a selectable Parallel Query Backend under FlowView Settings (default Legacy PHP Runner); when enabled, the parallel query parent runner (flowview_runner.php) delegates the whole query to the service over localhost instead of forking a PHP worker per shard, running the same cache-aware map-reduce (sharing parallel_database_query_shard_cache) and writing results back, with automatic fallback to the legacy runner when the service is unreachable
-
-* bug: Load the pre-1.3 report compatibility shims (reports_run/reports_queue/reports_log_and_notify) only when the running Cacti is older than 1.3, decided with cacti_version_compare(CACTI_VERSION, '1.3', ...) rather than a function_exists() probe; because the plugin's functions.php is included during plugin init before core loads lib/reports.php, the function_exists() check ran while the now-native helpers were still absent, declared the shims, and core then fatally redeclared reports_log_and_notify() on 1.3.0 develop builds (falls back to function_exists() only when the Cacti version helpers are unavailable, e.g. the unit tests)
-* security: Move the plugin's remaining inline event handlers to CSP-safe bindings so the pages no longer trip Cacti's Content-Security-Policy script-src-attr directive: confirmation Cancel buttons use the `cactiReturnTo` class, the database/filter selects are bound via change() in their ready blocks, and the main filter's sort/report selects and time-shift icons are bound in flowview_display_filter's ready block
-* dev: Replace the translation-template CI check that regenerated locales/po/cacti.pot and compared it with a diff-based gate (tests/bin/check-i18n-pot.php) requiring cacti.pot to be updated only when a pull request adds, removes, or modifies an i18n function call
-* issue: Prevent flow-capture from fatally erroring (TypeError in spl_object_hash) when flowview_db_check_reconnect() held a null shared-database handle; normalize a non-object connection to false so db_check_reconnect() checks and reconnects the default connection instead
-* dev: Measure CI coverage with xdebug instead of pcov so the plugin's own sources are instrumented (pcov auto-scopes to the Composer root and skipped cacti/plugins/, leaving the patch-coverage gate with nothing to measure)
-* dev: Enforce patch coverage of changed lines in CI and remove the inert COMPOSER_ROOT_VERSION env from the Pest step
-* dev: Pin the CI database image and shared test dependencies, disable persisted checkout credentials, and limit workflow permissions and runtime
+* security: Move the plugin's remaining inline event handlers to CSP-safe bindings
+* security: Add a version-safe CSP nonce (`plugin_flowview_csp_nonce()`) to every inline `<script>` tag
 * issue#110: Add NAT (postNAT source/destination IP and port) support to the core raw flow schema, filters, and DNS resolution; see flowview_upgrade_nat_columns.php to backfill existing partitions
-* security: Add a version-safe CSP nonce (`plugin_flowview_csp_nonce()`) to every inline `<script>` tag so pages stay compatible with Cacti's Content-Security-Policy nonce enforcement, while falling back cleanly on older Cacti releases that lack the `CactiSecureHeaders` class
-* chore: Harmonize CI workflow, issue/PR templates, and PHP-compatibility test structure with the shared Cacti plugin baseline
-* issue#261: Cacti 1.2.x flowview 5.0 update of 16.09 error on call db_check_reconnect inside flow_capture
 * issue#240: Call to db_table_exists is incorrect in some cases
+* issue#261: Cacti 1.2.x flowview 5.0 update of 16.09 error on call db_check_reconnect inside flow_capture
+* issue: Make bundled Net_DNS2 constructor nullability explicit for PHP 8.4 without changing accepted arguments
+* issue: Shorten the graph drilldown button title from 'View NetFlow Traffic In Range' to 'NetFlow' so it reads cleanly as a label in Cacti's graph action menu
+* issue: Create the plugin's config.php from config.php.dist for the flowview install step in the integration-test workflow (removed immediately afterwards so the unit tests still run with no config file), so the install no longer fails with "needs configuration"
+* issue: Load the pre-1.3 report compatibility shims (reports_run/reports_queue/reports_log_and_notify) only when the running Cacti is older than 1.3, decided with cacti_version_compare(CACTI_VERSION, '1.3', ...) rather than a function_exists() probe; because the plugin's functions.php is included during plugin init before core loads lib/reports.php, the function_exists() check ran while the now-native helpers were still absent, declared the shims, and core then fatally redeclared reports_log_and_notify() on 1.3.0 develop builds (falls back to function_exists() only when the Cacti version helpers are unavailable, e.g. the unit tests)
+* issue: Replace the translation-template CI check that regenerated locales/po/cacti.pot and compared it with a diff-based gate (tests/bin/check-i18n-pot.php) requiring cacti.pot to be updated only when a pull request adds, removes, or modifies an i18n function call
+* issue: Prevent flow-capture from fatally erroring (TypeError in spl_object_hash) when flowview_db_check_reconnect() held a null shared-database handle; normalize a non-object connection to false so db_check_reconnect() checks and reconnects the default connection instead
+* issue: Measure CI coverage with xdebug instead of pcov so the plugin's own sources are instrumented (pcov auto-scopes to the Composer root and skipped cacti/plugins/, leaving the patch-coverage gate with nothing to measure)
+* issue: Enforce patch coverage of changed lines in CI and remove the inert COMPOSER_ROOT_VERSION env from the Pest step
+* issue: Pin the CI database image and shared test dependencies, disable persisted checkout credentials, and limit workflow permissions and runtime
+* issue: Harmonize CI workflow, issue/PR templates, and PHP-compatibility test structure with the shared Cacti plugin baseline
 * issue: Killing processes does not occur when they are in D-State
-* chore: Bring all first-party PHP files to PHPStan level 8 with zero errors and add native parameter/return type declarations throughout (behavior-preserving)
-* fix: run_flow_query() used isset($sql_inner) as the statistical-vs-printed discriminator, but $sql_inner is always initialized so the check was always true and printed reports skipped the issue#110 NAT-safe SQL cleanup; corrected to a value check so printed reports now apply the NAT toggle
-* fix: Flow Collector "unable to open port" FATAL log always showed an empty error because $errno/$errstr were never populated; now sourced from socket_last_error()/socket_strerror()
-* fix: IRR database refresh no longer runs the present=0 cleanup DELETE when a database file download fails (the $files_broken guard was initialized but never engaged)
-* fix: Correct a duplicate 'src_rdomain' array key in the Source/Destination Root Domain report column map that overwrote the Destination Domain column
-* fix: Correct a stray "$" in the Source AS filter that read get_request_var() through an undefined variable-variable
-* fix: TCP flag validation regex used an unescaped delimiter making it always invalid, rejecting every TCP flag filter value
-* fix: load_data_for_filter()'s $start/$end were typed bool during the PHPStan pass, coercing the Unix timestamps passed by plugin_flowview_run_schedule() to true so scheduled reports queried around epoch second 1; restored the int|false contract
-* fix: The DNS Cache list (flowview_databases.php) and Schedules list (flowview_schedules.php) passed include_form=true to html_header_sort_checkbox() while their own chk form was already open, emitting nested forms that could misdirect checkbox submissions and bulk deletes; reverted to false
-* fix: IRR database refresh persisted the per-source serial after each individual file downloaded, so a later file failing left the refresh skipped on the next run (matching serial) with failed sections stuck at present=0; the serial is now advanced only after every file imports cleanly and every cleanup DELETE succeeds, batch insert failures are propagated through flowview_insert_irr_sections()/flowview_update_database(), and empty files are treated as broken
+* issue: Bring all first-party PHP files to PHPStan level 8 with zero errors and add native parameter/return type declarations throughout (behavior-preserving)
+* issue: run_flow_query() used isset($sql_inner) as the statistical-vs-printed discriminator, but $sql_inner is always initialized so the check was always true and printed reports skipped the issue#110 NAT-safe SQL cleanup; corrected to a value check so printed reports now apply the NAT toggle
+* issue: Flow Collector "unable to open port" FATAL log always showed an empty error because $errno/$errstr were never populated; now sourced from socket_last_error()/socket_strerror()
+* issue: IRR database refresh no longer runs the present=0 cleanup DELETE when a database file download fails (the $files_broken guard was initialized but never engaged)
+* issue: Correct a duplicate 'src_rdomain' array key in the Source/Destination Root Domain report column map that overwrote the Destination Domain column
+* issue: Correct a stray "$" in the Source AS filter that read get_request_var() through an undefined variable-variable
+* issue: TCP flag validation regex used an unescaped delimiter making it always invalid, rejecting every TCP flag filter value
+* issue: load_data_for_filter()'s $start/$end were typed bool during the PHPStan pass, coercing the Unix timestamps passed by plugin_flowview_run_schedule() to true so scheduled reports queried around epoch second 1; restored the int|false contract
+* issue: The DNS Cache list (flowview_databases.php) and Schedules list (flowview_schedules.php) passed include_form=true to html_header_sort_checkbox() while their own chk form was already open, emitting nested forms that could misdirect checkbox submissions and bulk deletes; reverted to false
+* issue: IRR database refresh persisted the per-source serial after each individual file downloaded, so a later file failing left the refresh skipped on the next run (matching serial) with failed sections stuck at present=0; the serial is now advanced only after every file imports cleanly and every cleanup DELETE succeeds, batch insert failures are propagated through flowview_insert_irr_sections()/flowview_update_database(), and empty files are treated as broken
+* feature: Add Cisco Apps (NBAR) reports for application-bearing NetFlow v9 exports, with IPv4/IPv6 application totals, conversations, paged connections, per-account capture/network options, recent-record status, first/last observation dates and a router setup guide
+* feature: Maintain reusable per-capture NBAR hourly/daily summaries through the existing Cacti poller, with raw partial-boundary fallback and a separate retention setting capped at 30 days
+* feature: Add an optional FlowView Query Service (a long-running Go service under service/flowview-query) as a selectable Parallel Query Backend under FlowView Settings (default Legacy PHP Runner); when enabled, the parallel query parent runner (flowview_runner.php) delegates the whole query to the service over localhost instead of forking a PHP worker per shard, running the same cache-aware map-reduce (sharing parallel_database_query_shard_cache) and writing results back, with automatic fallback to the legacy runner when the service is unreachable
 
 --- 5.0 ---
 

README.md

@@ -209,5 +209,91 @@ an issue.
 Got any ideas or complaints, please see the Cacti forums or GitHub for a
 resolution.
 
+
+## Cisco Apps (NBAR)
+
+Cisco Apps is a separate application report within FlowView. The current NBAR
+integration reads application IDs and names from NetFlow v9 data and options
+templates, including IPv4 and IPv6 records. IPFIX application decoding is not
+implemented by this integration. Existing FlowView reports retain their normal
+collector and report paths.
+
+Configure a FlowView listener for each export stream and configure the router to
+export application-bearing records to it. The report's Setup guide contains a
+Cisco 1100 Series example with placeholders for the collector, UDP port and router
+interfaces. Check Cisco documentation for the exact platform and IOS XE release.
+Creating a listener alone does not enable NBAR on the router.
+
+The report offers Top applications, Conversations and Connections. Filter by IP
+matches either endpoint. Conversations requires the account's Internal Networks
+settings to identify hosts and upload/download direction. Combine Options accepts
+two or three captures; leave them all unselected to save networks without combining.
+Choose one observation point per traffic path: overlapping LAN/WAN captures add
+both copies and can overcount traffic.
+
+STATUS describes recent retained application records for each selected capture,
+independently of historical report filters. A quiet capture can show no recent
+NBAR despite a working exporter. It is not a connectivity test.
+
+First seen and Last seen are the earliest/latest retained observation minutes in
+the selected period. They do not prove continuous activity, or give exact connection
+start/end times. Missing dates are shown as Unavailable. Average bandwidth divides
+data volume by the full selected period. Column sorting applies to the displayed
+page; resizing is saved in that browser.
+
+NBAR History Retention is separate from the existing FlowView Data Retention Policy.
+It offers 1, 3, 7, 14 or 30 days, with a seven-day default. Timespan choices and custom
+ranges respect this setting. The existing Cacti poller maintains summaries and
+performs bounded cleanup; no extra scheduled task is required. A shorter retention
+setting cannot be undone to recover deleted history. If polling stops, maintenance
+and cleanup stop too. Shared maintenance locks are scoped to the FlowView database.
+
+## Cisco Apps (NBAR) historical Connections
+
+Connections reads the automatically maintained daily history for each selected capture.
+Opening a report or changing its IP filter does not queue a new complete connection build.
+The normal FlowView poller maintains this history; no additional cron job is required.
+
+Pages contain up to 100 connections, ordered by total data volume across the selected
+captures and period. Indexed candidates are checked against every selected daily segment
+so byte, packet and record totals remain exact. Partial days use hourly or raw data.
+Previous, Next and a numeric page field provide navigation without a large dropdown or
+an expensive full-result count. Very deep pages or broadly tied results can still require
+more work; a shorter period or IP filter reduces that work.
+
+A small account-scoped session cache holds nearby pages for 120 seconds for past periods
+and 30 seconds for periods including current data. This is separate from retained history.
+The existing authenticated FlowView realm controls access; capture selections and filters are included in cache keys.
+
+## Reusable NBAR daily summaries
+
+Top Applications, Conversations and Connections share daily summaries built automatically
+per capture, independently of report filters and Cacti accounts. Calendar days run from
+local midnight to the following midnight in the Cacti server timezone, including daylight
+saving changes. Completed hours are added to today's summary in bounded transactions.
+Completed days are reused; late exports invalidate affected summaries and trigger a rebuild.
+Partial days and unpublished or changed intervals fall back to hourly summaries or raw records.
+IP, application and IP-version filters are applied to the preserved summary fields.
+
+Daily history follows Cisco Apps (NBAR) History Retention, currently selectable up to
+30 days. Retention is a rolling data window, not a new retention period beginning each time
+a report is opened. Daily summaries are shared source data and are reused for different
+periods, filters and users, subject to listener permissions. Connections reads pages from
+this history rather than preparing another full copy for every selection.
+
 -----------------------------------------------
 Copyright (c) 2004-2026 - The Cacti Group, Inc.
+
+NBAR daily summary materialization and observation-date backfill require MariaDB statement deadlines. On MySQL 8, these workers safely defer their DML rather than running without a bound; reports continue through hourly/raw sources. Exact counts use the MySQL SELECT execution-time hint.
+
+New NBAR records retain the actual exporter address through hourly and daily summaries. Legacy rows keep their counters with an unknown exporter; an exact single-address listener may still resolve their names, but earlier multi-exporter rows cannot be attributed retrospectively.
+
+NBAR maintenance is launched independently of the poller. A registered supervisor runs the PHP maintenance child with a 90-second process deadline. MariaDB hourly mutations also set and restore a verified 20-second statement deadline. MySQL hourly builds remain available only inside the supervised child; raw records remain the fallback until a checkpoint is published.
+
+NBAR timespan requests keep their session state separate from Flow Reports. Conversation totals exclude IPv4 and IPv6 multicast endpoints, and feed lookup failures are shown separately from empty captures. Retention maintenance drains additional batches across raw, hourly and daily tables within a shared ten-second cleanup budget, under the supervised worker deadline.
+
+Template decoding rejects zero-length fields and invalidates the affected cached template until a valid refresh arrives. Chart fallback labels are grouped by their complete expression, so applications without learned names keep separate slices.
+
+The NBAR template cache retains at most 256 templates / 8,192 field definitions in total and 64 templates / 2,048 fields per exporter, across all source IDs. Least recently used entries are evicted and inactive templates expire after 30 minutes (removed lazily on packet activity). Evicted templates are learned again from the next exporter refresh. Numeric data fields must fit unsigned 64-bit values; options scope fields are decoded separately and are not treated as counters.
+
+NBAR maintenance supervision uses PHP's direct argv-array process API rather than Cacti's version-dependent timeout command interface, so startup and hard child termination work on the minimum supported Cacti 1.2.29 and newer releases. Both output pipes are drained in bounded chunks.