Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Breaking Changes

* **role:fail2ban**: On hosts with Apache logs below `/var/log/httpd`, the role enables the Apache jails fail2ban ships (`apache-auth`, `apache-fakegooglebot`, `apache-modsecurity`, `apache-nohome`, `apache-noscript`, `apache-overflows`, `php-url-fopen`), unless `apache_httpd__mod_remoteip_internal_proxy` lists a proxy. On a host behind a reverse proxy that Apache logs as the client, such as LibreNMS behind a proxy, set `fail2ban__jails_apache_enabled: false`, otherwise the jails ban the proxy.
* **role:fail2ban**: The sshd jail also bans clients that connect without trying to log in, such as scanners (filter mode `aggressive`). Set `fail2ban__jail_sshd_mode: 'normal'` to keep the previous behaviour.
* **role:repo_baseos**: The Rocky Linux `security` repository downloads from `dl.rockylinux.org` instead of a mirror from the mirrorlist, so hot-fixes reach all hosts at the same time instead of whenever their mirror has synced. Hosts behind an egress filter need HTTPS access to `dl.rockylinux.org`.
* **role:system_update**: The regular update lane no longer sends a mail for every config-file stub (`*.rpmnew`, `*.rpmsave`, `*.dpkg-dist`, `*.ucf-dist`) it finds. Set `system_update__mail_new_configfiles_enabled: true` to keep these mails.
* **role:fail2ban**: The sshd jail bans on the port sshd listens on (`sshd__port`) instead of port 22, configurable with `fail2ban__jail_sshd_port`, and the portscan jail exempts that port instead of port 22 (`fail2ban__jail_portscan_allowed_ports`). On hosts where sshd listens on another port, sshd bans used to have no effect and now lock clients out, including admins who fail to log in three times within 15 minutes, and a denied connection attempt to port 22 now leads to a portscan ban. Add trusted networks to `fail2ban__jail_default_ignoreip` before updating, and list port 22 in `fail2ban__jail_portscan_allowed_ports` if it should stay exempt.
Expand Down Expand Up @@ -172,6 +174,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Changed

* **role:fail2ban**: A jail whose log file does not exist is skipped instead of keeping fail2ban from starting with all other jails (fail2ban 1.1.1 or newer).
* **role:apache_httpd, role:moodle, role:nextcloud**: vHosts log in the Apache standard format `combined` instead of `linuxfabrikio`, which fail2ban and log analysers read without further setup.
* **role:fail2ban**: The `apache-*` jails run with the values fail2ban ships for them (`maxretry`, `findtime`, `bantime`) instead of their own, and only point `logpath` at the logs of the apache_httpd role. For example, `apache-overflows` bans after 2 attempts for 10 minutes instead of after 1 for a day, and `apache-auth` after 5 failures instead of 3.
* **plugin:bitwarden_item**: Lookups are about ten times faster, which shortens runs against many hosts with credentials from Bitwarden, also with `--tags`, since every role validates its variables.
* **role:monitoring_plugins**: The source install prepares the plugins, the library and the Python dependencies on the controller once per release, or for `dev` once per commit, instead of once per host, keeps them in `/var/tmp/lfops` for later runs, and copies them to the hosts in parallel also under Mitogen, so runs against many hosts finish much faster. With ansible-navigator, mount `/var/tmp/lfops` into the execution environment to keep them between runs.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -34,30 +34,16 @@ fail2ban__filters__group_var:
failregex = ^.*molecule-fail2ban-probe from <HOST>$
journalmatch = _SYSTEMD_UNIT=sshd.service

# The jails fail2ban ships for Apache, which the role only points at the log files of the
# apache_httpd role. prepare.yml creates such a pair of log files.
# The Apache jails the role enables by itself need no entry here. apache-badbots and
# apache-botsearch are opt-in, so they are listed to prove their templates still load. A jail
# whose logpath matches no file proves that skip_if_nologs keeps fail2ban running without it.
fail2ban__jails__group_var:
- filename: 'z10-apache-auth'
state: 'present'
template: 'apache-auth'
- filename: 'z10-apache-badbots'
state: 'present'
template: 'apache-badbots'
- filename: 'z10-apache-botsearch'
state: 'present'
template: 'apache-botsearch'
- filename: 'z10-apache-fakegooglebot'
state: 'present'
template: 'apache-fakegooglebot'
- filename: 'z10-apache-nohome'
state: 'present'
template: 'apache-nohome'
- filename: 'z10-apache-noscript'
state: 'present'
template: 'apache-noscript'
- filename: 'z10-apache-overflows'
state: 'present'
template: 'apache-overflows'
- filename: 'z20-molecule-journal'
state: 'present'
template: 'raw'
Expand All @@ -69,3 +55,11 @@ fail2ban__jails__group_var:
findtime = 1m
maxretry = 3
port = 0:65535
- filename: 'z20-molecule-nolog'
state: 'present'
template: 'raw'
raw: |-
[molecule-nolog]
enabled = true
filter = apache-auth
logpath = /var/log/molecule-does-not-exist/*.log
4 changes: 4 additions & 0 deletions extensions/molecule/fail2ban/install/inventory/hosts.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,10 @@ lfops_fail2ban:

systems_under_test:
hosts:
# Stands in for an Apache behind a reverse proxy: with a proxy in
# apache_httpd__mod_remoteip_internal_proxy, the role leaves the Apache jails out.
rocky8-vm:
apache_httpd__mod_remoteip_internal_proxy:
- '192.0.2.1'
rocky9-vm:
rocky10-vm:
157 changes: 157 additions & 0 deletions extensions/molecule/fail2ban/install/verify.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,11 @@
# server reports back.
__molecule__fail2ban_sshd_bantime_seconds: 10800

# The role enables its Apache jails on a host with /var/log/httpd unless
# apache_httpd__mod_remoteip_internal_proxy lists a proxy, which install/inventory does for
# rocky8-vm only.
__molecule__fail2ban_apache_jails_expected: '{{ apache_httpd__mod_remoteip_internal_proxy | d([]) | length == 0 }}'

tasks:

# Floor check. On its own this would still pass with a dead server, so it is the start of the
Expand Down Expand Up @@ -61,9 +66,26 @@
loop_control:
label: '{{ item["item"] }}'

# The molecule-nolog jail of install/inventory reads a log file that does not exist. With
# skip_if_nologs from z00-defaults.conf the server skips it and runs everything else, which the
# running service above already shows; here the jail itself has to be absent.
- name: 'fail2ban-client status molecule-nolog'
ansible.builtin.command: 'fail2ban-client status molecule-nolog'
register: '__molecule__fail2ban_nolog_status_result'
changed_when: false
failed_when: false
check_mode: false

- name: 'Assert the server skipped the jail without a log file'
ansible.builtin.assert:
that: '__molecule__fail2ban_nolog_status_result["rc"] != 0'
fail_msg: 'the molecule-nolog jail runs: {{ __molecule__fail2ban_nolog_status_result["stdout"] }}'

# The apache-* jails are the ones fail2ban ships in jail.conf; the role only sets their
# logpath. The file list of the running jail shows the server reads the log of the virtual
# host that prepare.yml created, which the *error_log / *access_log of jail.conf would miss.
# apache-badbots and apache-botsearch come from install/inventory and run everywhere, the
# others only where the role enables its Apache jails.
- name: 'fail2ban-client status {{ item["jail"] }}'
ansible.builtin.command: 'fail2ban-client status {{ item["jail"] }}'
register: '__molecule__fail2ban_apache_status_result'
Expand All @@ -75,16 +97,22 @@
log: 'error'
- jail: 'apache-badbots'
log: 'access'
inventory: true
- jail: 'apache-botsearch'
log: 'error'
inventory: true
- jail: 'apache-fakegooglebot'
log: 'access'
- jail: 'apache-modsecurity'
log: 'error'
- jail: 'apache-nohome'
log: 'error'
- jail: 'apache-noscript'
log: 'error'
- jail: 'apache-overflows'
log: 'error'
- jail: 'php-url-fopen'
log: 'access'
loop_control:
label: '{{ item["jail"] }}'

Expand All @@ -97,6 +125,19 @@
loop: '{{ __molecule__fail2ban_apache_status_result["results"] }}'
loop_control:
label: '{{ item["item"]["jail"] }}'
when:
- 'item["item"]["inventory"] | d(false) or __molecule__fail2ban_apache_jails_expected | bool'

- name: 'Assert the Apache jails are left out behind a reverse proxy'
ansible.builtin.assert:
that: 'item["rc"] != 0'
fail_msg: 'the {{ item["item"]["jail"] }} jail runs, although apache_httpd__mod_remoteip_internal_proxy lists a proxy'
loop: '{{ __molecule__fail2ban_apache_status_result["results"] }}'
loop_control:
label: '{{ item["item"]["jail"] }}'
when:
- 'not item["item"]["inventory"] | d(false)'
- 'not __molecule__fail2ban_apache_jails_expected | bool'

# The role sets no maxretry or bantime for these jails, so the server has to run with the
# values of jail.conf: maxretry 2 for apache-overflows, bantime 48h for apache-badbots.
Expand All @@ -115,6 +156,8 @@
expected: 2
loop_control:
label: '{{ item["jail"] }} {{ item["option"] }}'
when:
- 'item["jail"] == "apache-badbots" or __molecule__fail2ban_apache_jails_expected | bool'

- name: 'Assert the apache-* jails use the values of jail.conf'
ansible.builtin.assert:
Expand All @@ -123,6 +166,8 @@
loop: '{{ __molecule__fail2ban_apache_option_result["results"] }}'
loop_control:
label: '{{ item["item"]["jail"] }} {{ item["item"]["option"] }}'
when:
- 'item["skipped"] is not defined'

# A path traversal attempt that Apache rejects with `AH10244: invalid URI path`, written to the
# log of the virtual host twice, the maxretry of jail.conf. apache-overflows only knows AH10244
Expand All @@ -136,6 +181,8 @@
done >> /var/log/httpd/molecule.example.com-error.log
changed_when: false
check_mode: false
when:
- '__molecule__fail2ban_apache_jails_expected | bool'

- name: 'fail2ban-client status apache-overflows'
ansible.builtin.command: 'fail2ban-client status apache-overflows'
Expand All @@ -146,17 +193,23 @@
retries: 10
delay: 3
failed_when: false
when:
- '__molecule__fail2ban_apache_jails_expected | bool'

- name: 'Assert apache-overflows banned the source of the path traversal attempts'
ansible.builtin.assert:
that: '"198.51.100.20" in __molecule__fail2ban_overflows_status_result["stdout"]'
fail_msg: 'apache-overflows did not ban 198.51.100.20: {{ __molecule__fail2ban_overflows_status_result["stdout"] }}'
when:
- '__molecule__fail2ban_apache_jails_expected | bool'

# verify.yml runs twice, and the second run has to see the ban happen again.
- name: 'fail2ban-client set apache-overflows unbanip 198.51.100.20'
ansible.builtin.command: 'fail2ban-client set apache-overflows unbanip 198.51.100.20'
changed_when: false
check_mode: false
when:
- '__molecule__fail2ban_apache_jails_expected | bool'

# The wordpress-login filter ships with the role defaults, but its jail reads Apache logs this
# host does not have. So the deployed filter is fed sample lines through fail2ban-regex, which
Expand Down Expand Up @@ -194,6 +247,95 @@
loop_control:
label: '{{ item["item"]["label"] }}'

# The filters of the Apache jails, fed the lines Apache 2.4 writes for the requests they are
# meant for. Measured on Rocky 8, 9 and 10 (httpd 2.4.37, 2.4.62, 2.4.63, mod_security 2.9)
# against real requests and rewritten with documentation addresses. A filter that no longer
# matches the current wording of Apache fails here, independent of the jails this host runs.
- name: 'fail2ban-regex {{ item["filter"] }} against {{ item["label"] }}'
ansible.builtin.command:
argv:
- 'fail2ban-regex'
- '{{ item["line"] }}'
- '/etc/fail2ban/filter.d/{{ item["filter"] }}.conf'
register: '__molecule__fail2ban_apache_regex_result'
changed_when: false
check_mode: false
loop:
- label: 'an unknown user (AH01618)'
filter: 'apache-auth'
line: '[Wed Nov 18 12:30:05.123456 2026] [auth_basic:error] [pid 1234:tid 1234] [client 198.51.100.30:51234] AH01618: user bob not found: /protected/'
matched: 1
- label: 'a wrong password (AH01617)'
filter: 'apache-auth'
line: '[Wed Nov 18 12:30:05.123456 2026] [auth_basic:error] [pid 1234:tid 1234] [client 198.51.100.30:51234] AH01617: user alice: authentication failure for "/protected/": Password Mismatch'
matched: 1
- label: 'a request denied by Require (AH01630)'
filter: 'apache-auth'
line: '[Wed Nov 18 12:30:05.123456 2026] [authz_core:error] [pid 1234:tid 1234] [client 198.51.100.30:51234] AH01630: client denied by server configuration: /var/www/html/denied'
matched: 1
- label: 'a request ModSecurity denied (Rocky 10)'
filter: 'apache-modsecurity'
line: '[Wed Nov 18 12:30:05.123456 2026] [security2:error] [pid 1234:tid 1234] [client 198.51.100.31:51234] ModSecurity: Access denied with code 403 (phase 2). Operator GE matched 5 at TX:blocking_inbound_anomaly_score. [file "/etc/httpd/modsecurity.d/activated_rules/REQUEST-949-BLOCKING-EVALUATION.conf"] [line "222"] [id "949110"] [msg "Inbound Anomaly Score Exceeded (Total Score: 5)"] [hostname "192.0.2.5"] [uri "/"] [unique_id "aW1vbGVjdWxl"]'
matched: 1
- label: 'a request ModSecurity denied (Rocky 8 and 9)'
filter: 'apache-modsecurity'
line: '[Wed Nov 18 12:30:05.123456 2026] [:error] [pid 1234:tid 1234] [client 198.51.100.31:51234] [client 198.51.100.31] ModSecurity: Access denied with code 403 (phase 2). Operator GE matched 5 at TX:anomaly_score. [file "/etc/httpd/modsecurity.d/activated_rules/REQUEST-949-BLOCKING-EVALUATION.conf"] [line "80"] [id "949110"] [msg "Inbound Anomaly Score Exceeded (Total Score: 5)"] [hostname "192.0.2.5"] [uri "/"] [unique_id "aW1vbGVjdWxl"]'
matched: 1
- label: 'a probe for a home directory'
filter: 'apache-nohome'
line: '[Wed Nov 18 12:30:05.123456 2026] [core:info] [pid 1234:tid 1234] [client 198.51.100.32:51234] AH00128: File does not exist: /var/www/html/~root/'
matched: 1
- label: 'a missing favicon'
filter: 'apache-nohome'
line: '[Wed Nov 18 12:30:05.123456 2026] [core:info] [pid 1234:tid 1234] [client 198.51.100.40:51234] AH00128: File does not exist: /var/www/html/favicon.ico'
matched: 0
- label: 'a missing PHP script (AH01071)'
filter: 'apache-noscript'
line: "[Wed Nov 18 12:30:05.123456 2026] [proxy_fcgi:error] [pid 1234:tid 1234] [client 198.51.100.33:51234] AH01071: Got error 'Primary script unknown'"
matched: 1
- label: 'a missing ASP script (AH00128)'
filter: 'apache-noscript'
line: '[Wed Nov 18 12:30:05.123456 2026] [core:info] [pid 1234:tid 1234] [client 198.51.100.33:51234] AH00128: File does not exist: /var/www/html/missing.asp'
matched: 1
- label: 'a missing CGI script (AH01264, Rocky 9)'
filter: 'apache-noscript'
line: '[Wed Nov 18 12:30:05.123456 2026] [cgid:error] [pid 1234:tid 1234] [client 198.51.100.33:51234] AH01264: stderr from /var/www/html/cgi-bin/missing.sh: script not found or unable to stat'
matched: 1
- label: 'a missing favicon'
filter: 'apache-noscript'
line: '[Wed Nov 18 12:30:05.123456 2026] [core:info] [pid 1234:tid 1234] [client 198.51.100.40:51234] AH00128: File does not exist: /var/www/html/favicon.ico'
matched: 0
- label: 'a path traversal attempt (AH00126, Rocky 8)'
filter: 'apache-overflows'
line: '[Wed Nov 18 12:30:05.123456 2026] [core:error] [pid 1234:tid 1234] [client 198.51.100.34:51234] AH00126: Invalid URI in request GET /../../../etc/passwd HTTP/1.1'
matched: 1
- label: 'a path traversal attempt (AH10244, Rocky 9 and 10)'
filter: 'apache-overflows'
line: '[Wed Nov 18 12:30:05.123456 2026] [core:error] [pid 1234:tid 1234] [client 198.51.100.34:51234] AH10244: invalid URI path (/%2e%2e/%2e%2e/etc/passwd)'
matched: 1
- label: 'a remote URL passed to a PHP script'
filter: 'php-url-fopen'
line: '198.51.100.35 - - [18/Nov/2026:12:30:05 +0100] "GET /index.php?file=http://www.example.com/x.txt HTTP/1.1" 404 196 "-" "curl/8"'
matched: 1
- label: 'an encoded redirect URL'
filter: 'php-url-fopen'
line: '198.51.100.40 - - [18/Nov/2026:12:30:05 +0100] "GET /?redirect_uri=https%3A%2F%2Fwww.example.com%2Fcb HTTP/1.1" 200 3 "-" "curl/8"'
matched: 0
- label: 'a client claiming to be Googlebot'
filter: 'apache-fakegooglebot'
line: '198.51.100.36 - - [18/Nov/2026:12:30:05 +0100] "GET / HTTP/1.1" 200 3 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"'
matched: 1
loop_control:
label: '{{ item["filter"] }}: {{ item["label"] }}'

- name: 'Assert the filters of the Apache jails match what Apache logs'
ansible.builtin.assert:
that: 'item["stdout"] is search("Lines: 1 lines, 0 ignored, " ~ item["item"]["matched"] ~ " matched")'
fail_msg: '{{ item["item"]["filter"] }}, {{ item["item"]["label"] }}: {{ item["stdout"] }}'
loop: '{{ __molecule__fail2ban_apache_regex_result["results"] }}'
loop_control:
label: '{{ item["item"]["filter"] }}: {{ item["item"]["label"] }}'

# Same approach for the portscan filter, whose jail reads the kernel log of a firewall this
# host does not run. The filter gets its ports and addresses from the jail, so the sample
# lines are fed with allowed port 22, what the role default gives on a host with sshd on port
Expand Down Expand Up @@ -279,6 +421,21 @@
that: '__molecule__fail2ban_sshd_bantime_result["stdout"] | int == __molecule__fail2ban_sshd_bantime_seconds'
fail_msg: 'the sshd bantime is {{ __molecule__fail2ban_sshd_bantime_result["stdout"] }}s, expected {{ __molecule__fail2ban_sshd_bantime_seconds }}s ({{ fail2ban__jail_sshd_bantime }})'

# The aggressive mode adds the regexes of the ddos and extra modes, for example scanners that
# break off the key exchange. Neither appears in the failregex of the default normal mode.
- name: 'fail2ban-client get sshd failregex'
ansible.builtin.command: 'fail2ban-client get sshd failregex'
register: '__molecule__fail2ban_sshd_failregex_result'
changed_when: false
check_mode: false

- name: 'Assert the sshd jail runs the filter in aggressive mode'
ansible.builtin.assert:
that:
- '"kex_exchange_identification" in __molecule__fail2ban_sshd_failregex_result["stdout"]'
- '"authentication methods available" in __molecule__fail2ban_sshd_failregex_result["stdout"]'
fail_msg: 'the sshd jail does not run in aggressive mode: {{ __molecule__fail2ban_sshd_failregex_result["stdout"] }}'

# The port is what the banaction later puts into the iptables rule. The rule itself only
# appears with the first ban (iptables.conf has family-conditional sections, so fail2ban sets
# actionstart_on_demand), so the port is read from the running action instead. Without the
Expand Down
Loading
Loading