diff --git a/CHANGELOG.md b/CHANGELOG.md index 09bc11c7..64b0677e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -20,6 +20,11 @@ kept as history. This entry is the one to read if you are coming from **1.15.1** ### Added +- **`SNAP_DIFF_ACCEPT_SCREENSHOTS=1` re-records changed baselines for one run.** The variable + selects `record = :all` for the run that sees it, outranking `record =` and `fail_if_new`; a + `record:` on the screenshot still outranks it. `1` and `true` turn it on; any other value is the + same as not setting it. Unset, the mode reads as before (`:none` under CI, `:once` off it). + - **AI triage (optional, offline-first, advisory by default).** `SnapDiff::Reporters::AISimple` classifies every failed comparison as `flaky` / `intentional` / `real_bug`, logs one line per diff, quotes the verdict in the failure message, and @@ -341,6 +346,10 @@ One removal 2.0 cannot warn about, written down instead: the legacy `LOADED_DRIV rollback) ### Changed +- **`SNAP_DIFF_ACCEPT_SCREENSHOTS=1` lets `record: :all` run under CI.** A job that re-records + baselines sets the variable and nothing else: no `CI: ""`, no `continue-on-error`, and a failing + unrelated test still fails the job. A `record = :all` that comes from code is still refused under + CI when the variable is not set. The update-baselines recipe in `docs/ci-integration.md` uses it. - **`record: :all` now compares before re-recording and the report shows what changed.** A screenshot that differs from its committed baseline is accepted as the new baseline (it never fails the run) and leaves the usual `.base.png` and diff/heatmap files, diff --git a/README.md b/README.md index c8e6bc8c..699b2032 100644 --- a/README.md +++ b/README.md @@ -199,18 +199,11 @@ diff next to the code that caused it. ### Accepting many at once -One failure at a time is fine for one screenshot. After a redesign that changed forty, set the -**record mode** to `:all` and re-record them in a single run: - -```ruby -# test_helper.rb -# Set the mode only when asked: an explicit mode, even :once, outranks the -# :none-under-CI default and would stop CI from failing on a missing baseline. -SnapDiff.config.record = :all unless ENV["ACCEPT_SCREENSHOTS"].to_s.empty? -``` +One failure at a time is fine for one screenshot. After a redesign that changed forty, re-record +them in a single run: ```bash -ACCEPT_SCREENSHOTS=1 bin/rails test:system +SNAP_DIFF_ACCEPT_SCREENSHOTS=1 bin/rails test:system # [snap_diff] 212 verified, 40 changed (40 re-recorded), 0 new (not verified). git status # forty modified baselines, and nothing else @@ -218,11 +211,12 @@ git add doc/screenshots/ git commit -m "chore: re-record baselines after the checkout redesign" ``` -`:all` compares every capture to its committed baseline like `:once` does, but a screenshot that -differs is accepted as the new baseline instead of failing the test — the report lists what changed, -and `git status` is the review surface — look at the images before you commit. It **refuses to run under CI**, -where there would be nobody to look. The other modes are `:once` (the default) and `:none` -(strict — a missing baseline always fails); see +The variable puts the run in the `:all` **record mode**: every screenshot is still compared to its +committed baseline, but one that differs becomes the new baseline instead of failing the test. The +report lists what changed and `git status` is the review surface — look at the images before you +commit. To do the same on CI, set the variable on a dedicated job; see +[CI integration](docs/ci-integration.md#update-baselines-in-ci). The other modes are `:once` (the +default off CI) and `:none` (the default under CI; a missing baseline always fails); see [Record modes](docs/configuration.md#record-modes--accepting-changes). ## Web UI for Reviewing Screenshot Changes diff --git a/docs/UPGRADING.md b/docs/UPGRADING.md index 44a57e92..7a22f6bf 100644 --- a/docs/UPGRADING.md +++ b/docs/UPGRADING.md @@ -339,12 +339,9 @@ process, from the point you set it: **Nothing changes if you set none of them.** With no `record` line, `record` reads back as `:none` under CI and `:once` off it — exactly what `fail_if_new` already did, sniff and all. The missing-baseline default is deliberately unchanged; `:none` is how you opt into strictness -explicitly. **Precedence:** an explicitly set mode outranks `fail_if_new`, and `fail_if_new` -decides only when no mode was set — the same rule `fail_if_new` itself has over the `CI` sniff. - -`record = :all` **refuses to run under CI**: it accepts every difference by design, so left in a -committed config file it would be a build that never fails. See -[Record modes](configuration.md#record-modes--accepting-changes). +explicitly. An explicitly set mode outranks `fail_if_new`, and `SNAP_DIFF_ACCEPT_SCREENSHOTS=1` +outranks both for the run that sets it; the full chain is under +[Precedence](configuration.md#precedence). #### Silent by design diff --git a/docs/ci-integration.md b/docs/ci-integration.md index 1d37c886..c4e11e1e 100644 --- a/docs/ci-integration.md +++ b/docs/ci-integration.md @@ -243,9 +243,8 @@ jobs: - name: Record baselines run: bin/rails test:system - continue-on-error: true # the run fails by design; it rewrites the baselines env: - CI: "" # see below — without this, a NEW screenshot is never written + SNAP_DIFF_ACCEPT_SCREENSHOTS: "1" # record: :all for this job only - name: Commit updated baselines run: | @@ -258,23 +257,26 @@ jobs: -> **Set the record mode, or clear `CI`, to let this job record a *new* baseline.** With -> nothing set, a missing baseline fails whenever `ENV["CI"]` is set and non-empty — so on a -> stock GitHub Actions runner a screenshot with no committed baseline raises `No existing -> screenshot found for …`. `SnapDiff.config.record = :once` -> (or clearing `CI` for this one step, or the older `SnapDiff.config.fail_if_new = false`) -> records new baselines instead. *Changed* baselines are rewritten either way; only new ones -> need this. See [Record modes](configuration.md#record-modes--accepting-changes). -> -> The screenshot itself **is** written before the raise ([`screenshot_matcher.rb`](https://github.com/snap-diff/snap_diff-capybara/blob/master/lib/snap_diff/screenshot_matcher.rb) -> — `capture_screenshot` precedes `fail_if_new_screenshot`), so the `git add` the message names -> is a command you can actually run. But the raise still fails the test it happened in, and a red -> job usually never reaches the commit step — which is why a recording job sets the mode rather -> than relying on the files being there. -> -> **`record = :all` is not the mode for this job** — it refuses to run under CI, because it -> would accept every *changed* rendering unreviewed and never fail. `:once` records what is new and -> keeps comparing everything that already has a baseline. +`SNAP_DIFF_ACCEPT_SCREENSHOTS` puts the run in the `:all` record mode: every screenshot is +compared to its committed baseline, one that differs becomes the new baseline, one with no +baseline yet is recorded, and the run exits green with a summary of what it re-recorded +(`3 changed (3 re-recorded)`). A failing test that has nothing to do with screenshots still fails +the job. See [Record modes](configuration.md#record-modes--accepting-changes) for the mode itself. + +Set the variable on this job only. The gem cannot tell a variable set on one job from one exported +to every job, and on a verification job it would accept every change. Do not put +`record = :all` in `test_helper.rb` for the same reason; under CI the gem refuses a `:all` that +comes from code. + +Record in the same environment that verifies (OS, browser, fonts). A baseline recorded elsewhere +is often worthless, which is why this job runs on CI at all. + +Two things this minimal recipe does not do: + +- It keeps the committed PNGs, not the HTML report or the diff images. Add an + [upload step](#2-reusable-composite-action-recommended) without an `if: failure()` guard if you want them. +- A push made with the default `GITHUB_TOKEN` does not start the ordinary verification + workflow. Review the images in the pushed commit, then run the normal check against it. **How it works:** 1. Go to Actions → "Update Screenshot Baselines" → "Run workflow" diff --git a/docs/configuration.md b/docs/configuration.md index a7d6591f..71407ef0 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -68,20 +68,18 @@ SnapDiff.config.record = :once # what a local run already does; see Precedence ### `:all` — the bulk-accept verb After an intentional redesign that changed forty screenshots, re-record them in one run rather -than accepting them one failure at a time: - -```ruby -# test_helper.rb. There is no CLI flag — the gem has no runner to hang one on — -# so gate it on an environment variable of your own if you want one. Set the -# mode only when asked: an explicit mode, even :once, outranks the -# :none-under-CI default and would stop CI from failing on a missing baseline. -SnapDiff.config.record = :all unless ENV["ACCEPT_SCREENSHOTS"].to_s.empty? -``` +than accepting them one failure at a time. Set `SNAP_DIFF_ACCEPT_SCREENSHOTS` for that run: ```bash -ACCEPT_SCREENSHOTS=1 bin/rails test:system # with the line above in test_helper.rb +SNAP_DIFF_ACCEPT_SCREENSHOTS=1 bin/rails test:system ``` +The variable selects `:all` for the run that sees it. `1` and `true` (any case) turn it on; any +other value, including `0`, `false` and empty, is the same as not setting it. It wins over +`record =` and `fail_if_new` (see Precedence below), so a suite that pins `record = :none` accepts +with the same command. Do not write `record = :all` in `test_helper.rb` instead: a mode written +there applies to every run, including the ones that are supposed to fail. + Every screenshot is compared to its committed baseline, as under `:once`. One that differs is kept as the new baseline instead of failing the test, and leaves the usual `.base.png` and diff files (ignored by the recommended `.gitignore`); one that matches changes nothing. So `git status` lists @@ -95,14 +93,14 @@ At the end of the run the gem names what it accepted: [snap_diff] record: :all compared 3 screenshots, found them different and accepted them as the new baseline: checkout/cart, checkout/payment, checkout/review. `git add` the file(s) to keep the change, `git restore -- ` to discard it. The `.base.` and diff files left beside each one are covered by the recommended .gitignore. ``` -> **`:all` refuses to run under CI** (when `ENV['CI']` is set to a non-empty value). It accepts -> every difference by design, so a mode left in a committed config file would buy you a build that -> never fails, with the "recorded" screenshots discarded when the runner -> is torn down. Re-record locally, where you can look at the result. +> **Under CI (`ENV['CI']` non-empty), `:all` runs only when `SNAP_DIFF_ACCEPT_SCREENSHOTS` is +> set.** When the variable is not set, a `:all` written in code (a config line or a per-screenshot +> option) raises there, because a committed `:all` would make every build pass. When it is set, the +> whole run records, whatever the code says. To re-record baselines on CI, set +> the variable on a dedicated job: see [Update baselines in CI](ci-integration.md#update-baselines-in-ci). > -> A CI job that needs to record screenshots with **no baseline yet** does not need `:all`: -> `record = :once` records those and still compares everything that has a baseline. See -> [CI integration](ci-integration.md). +> A CI job that only needs to record screenshots with **no baseline yet** does not need `:all`: +> `record = :once` records those and still compares everything that has a baseline. ### Per screenshot @@ -112,9 +110,16 @@ assert_matches_screenshot "flaky_widget", record: :none # this one must alread ### Precedence -**An explicitly set mode outranks `fail_if_new`; `fail_if_new` decides only when no mode was set.** -That is the same rule `fail_if_new` itself has over the `CI` sniff — explicit outranks implicit, all -the way down. Per screenshot outranks the config; the config outranks `fail_if_new`. +First match wins: + +1. `record:` on the screenshot +2. `SNAP_DIFF_ACCEPT_SCREENSHOTS=1`, which means `:all` +3. `SnapDiff.config.record =` +4. `fail_if_new`, when set +5. the default: `:none` under CI, `:once` off it + +A setting written for one screenshot beats the run's variable, the run's variable beats the +suite's configuration, and an explicitly set mode beats the legacy `fail_if_new` boolean. With **nothing** set, `record` reads back as `:none` under CI and `:once` off it — which is exactly what `fail_if_new` has always done, so a suite that never mentions `record` behaves as it always @@ -126,9 +131,11 @@ laptop is often worthless on another OS. `:none` makes strictness an explicit ch |-----------|----------------|------------------| | nothing, off CI | `:once` | recorded | | nothing, under CI | `:none` | fails | +| nothing, `SNAP_DIFF_ACCEPT_SCREENSHOTS=1` | `:all` | recorded; a changed baseline is re-recorded | | `record = :once` | `:once` | recorded, on CI too | | `record = :none` | `:none` | fails, off CI too | | `record = :once`, `fail_if_new = true` | `:once` | recorded — the mode wins | +| `record = :none`, `SNAP_DIFF_ACCEPT_SCREENSHOTS=1` | `:all` | recorded — the variable wins | | `record = nil`, `fail_if_new = true` | `:none` | fails — nil hands it back | A misspelt mode raises `ArgumentError` at the point you set it, rather than reading back as diff --git a/lib/snap_diff/config.rb b/lib/snap_diff/config.rb index 3ff9a5ab..432b2f57 100644 --- a/lib/snap_diff/config.rb +++ b/lib/snap_diff/config.rb @@ -107,6 +107,11 @@ def self.ci? !ENV["CI"].to_s.empty? end + # The per-run spelling of record = :all; see {#record}. + def self.accept_screenshots? + Removal.truthy_env?("SNAP_DIFF_ACCEPT_SCREENSHOTS") + end + # shift_distance_limit, driver and the three older new-screenshot # booleans are excluded from the generated writers and hand written # below (they announce their 2.1 removal, or validate); fail_if_new and @@ -200,13 +205,17 @@ def fail_on_difference=(value) # +:all+:: +:once+ that accepts instead of fails. Every baseline is # compared; a screenshot that differs is kept as the new # baseline and never raises -- the bulk-accept verb, for the - # redesign that changed forty screenshots at once. Refused - # under CI (see {SnapDiff::ScreenshotMatcher}). + # redesign that changed forty screenshots at once. Written in + # code, it is refused under CI; the variable below is how a + # CI job records (see {SnapDiff::ScreenshotMatcher}). # - # PRECEDENCE: an explicitly set mode outranks +fail_if_new+; nil means - # nobody said, and only then does +fail_if_new+ answer -- so a setup - # with no +record+ line behaves EXACTLY as it did before this setting - # existed, CI sniff and all. +:none+ is the mode spelling of + # PRECEDENCE: the per-screenshot +record:+ option outranks everything. + # Then SNAP_DIFF_ACCEPT_SCREENSHOTS=1 selects +:all+ for this one + # run: the bulk-accept verb is a per-run decision, so the run's own + # intent beats a mode written into test_helper.rb, which lasts forever. + # Then an explicit +record =+, then +fail_if_new+ -- so a setup with no + # variable behaves EXACTLY as it did before this setting existed, CI + # sniff and all. +:none+ is the mode spelling of # fail_if_new = true and +:once+ of = false, which is # why the fallback can express today's behaviour without a special case # anywhere downstream: the matcher branches on the mode alone. @@ -217,6 +226,8 @@ def fail_on_difference=(value) # worthless across OS. +:none+ makes strictness an explicit choice # instead; that is the whole point of having the mode. def record + return :all if Config.accept_screenshots? + @record || (fail_if_new ? :none : :once) end diff --git a/lib/snap_diff/removal.rb b/lib/snap_diff/removal.rb index e5add3c2..4f3cb473 100644 --- a/lib/snap_diff/removal.rb +++ b/lib/snap_diff/removal.rb @@ -84,6 +84,16 @@ def self.unknown_option(key) "ArgumentError for it. Check the spelling against the option list in docs/configuration.md." end + # Is the environment variable +name+ switched on? Truthy is "1" or "true" + # (any case); anything else, empty included, is off. The one rule for the + # gem's own on/off switches, so a value that stops failures is never + # enabled by "0" or "false" (+CI+ keeps its non-empty rule: it is not + # ours, and "false" is how some runners spell it). Lives here because this + # file loads under every entry point, Config included. + def self.truthy_env?(name) + %w[1 true].include?(ENV[name].to_s.downcase) + end + MUTEX = Mutex.new @seen = {} @suppressed = false @@ -147,13 +157,7 @@ class << self # either via the {silence_deprecations} accessor or the # SNAP_DIFF_SILENCE_DEPRECATIONS env var (truthy = "1"/"true"). def silence_deprecations? - !!silence_deprecations || truthy_env?(ENV["SNAP_DIFF_SILENCE_DEPRECATIONS"]) - end - - private - - def truthy_env?(value) - %w[1 true].include?(value.to_s.downcase) + !!silence_deprecations || Removal.truthy_env?("SNAP_DIFF_SILENCE_DEPRECATIONS") end end end diff --git a/lib/snap_diff/screenshot_matcher.rb b/lib/snap_diff/screenshot_matcher.rb index c35f5a51..62547d76 100644 --- a/lib/snap_diff/screenshot_matcher.rb +++ b/lib/snap_diff/screenshot_matcher.rb @@ -74,23 +74,27 @@ def resolve_record_mode(per_screenshot) end # `:all` accepts every differing rendering as the new baseline -- that is - # the feature. On CI there is nobody to review the result and the - # re-recorded files go away with the box, so a mode left in a committed - # config file buys a build that never fails, forever. That is precisely - # how Percy goes green on a job that lost its token; refuse it rather - # than shipping our own version. + # the feature. A mode written in code (a config line, a per-screenshot + # option) applies to every run, so left on CI it buys a build that never + # fails, forever. That is precisely how Percy goes green on a job that + # lost its token; refuse it rather than shipping our own version. # - # A CI job that must record NEW baselines does not need `:all` at all: - # `record: :once` records them and still compares everything that has a - # baseline (docs/ci-integration.md). + # The per-run variable is the operator asking for this run to record, so + # a deliberate recording job (docs/ci-integration.md) gets through. With + # it set, the whole run is `:all` already; where the mode was written no + # longer matters, so the guard looks only at the variable. + # + # A CI job that must record only NEW baselines does not need `:all`: + # `record: :once` records them and still fails on everything that differs. def refuse_bulk_record_under_ci! return unless SnapDiff::Config.ci? + return if SnapDiff::Config.accept_screenshots? raise SnapDiff::ExpectationNotMet.new(<<~ERROR.chomp, caller) - `record: :all` re-records every screenshot that differs from its baseline without failing, so it refuses to run under CI (ENV["CI"] is set). - No difference could fail the build and the re-recorded screenshots would be discarded with the runner. - Re-record locally, review the result, and commit the screenshots. - To let a CI job record screenshots that have no baseline yet: SnapDiff.config.record = :once + `record: :all` re-records every screenshot that differs from its baseline without failing, and it was set in code, so it refuses to run under CI (ENV["CI"] is set). + Left in committed config, no difference could fail this build, ever. + To record on CI for one run: SNAP_DIFF_ACCEPT_SCREENSHOTS=1 + To record only the screenshots that have no baseline yet: SnapDiff.config.record = :once ERROR end diff --git a/test/fixtures/summary_line_case.rb b/test/fixtures/summary_line_case.rb index cb3614e0..683548d3 100644 --- a/test/fixtures/summary_line_case.rb +++ b/test/fixtures/summary_line_case.rb @@ -12,6 +12,7 @@ # SNAP_IMAGES -- directory holding the fixture PNGs a.png / b.png # SNAP_CASES -- comma-separated subset of verified,changed,new (may be empty) # SNAP_RECORD -- optional record mode for the whole run (once, none, all) +# SNAP_FAIL_ONE -- when set, adds one unrelated test that fails require "minitest/autorun" require "snap_diff/integrations/minitest" # The HTML report is a FEATURE and stays opt-in; the summary line is core @@ -69,4 +70,10 @@ class SummaryLineCase < Minitest::Test end end end + + if ENV["SNAP_FAIL_ONE"] + def test_unrelated + assert false, "unrelated failure" + end + end end diff --git a/test/integration/summary_line_test.rb b/test/integration/summary_line_test.rb index 04999a41..d3aa7e6d 100644 --- a/test/integration/summary_line_test.rb +++ b/test/integration/summary_line_test.rb @@ -119,13 +119,48 @@ class SummaryLineTest < ActiveSupport::TestCase assert_includes out, "[snap_diff] 2 verified, 1 changed (1 re-recorded), 0 new (not verified)." end + # The documented accept run: the variable on CI, no `record` line anywhere. What differs is + # accepted and kept as the baseline, so a run with nothing else wrong exits green, which is + # what lets the recipe reach its commit step -- and an unrelated failure still fails it. + test "SNAP_DIFF_ACCEPT_SCREENSHOTS on CI accepts what differs and the run succeeds" do + out, status, report, baseline = accept_run_on_ci + + assert status.success?, out + assert_accepted_on_ci(out, report, baseline) + end + + test "SNAP_DIFF_ACCEPT_SCREENSHOTS on CI still fails the run for an unrelated failing test" do + out, status, report, baseline = accept_run_on_ci("SNAP_FAIL_ONE" => "1") + + refute status.success?, out + assert_includes out, "unrelated failure" + assert_match(/1 failures, 0 errors/, out) + assert_accepted_on_ci(out, report, baseline) + end + private + def accept_run_on_ci(extra_env = {}) + baseline = nil + env = {"CI" => "1", "SNAP_DIFF_ACCEPT_SCREENSHOTS" => "1"}.merge(extra_env) + out, status, report = run_case("verified,changed", env: env) do |repo| + baseline = File.binread("#{repo}/screenshots/changed.png") + end + [out, status, report, baseline] + end + + def assert_accepted_on_ci(out, report, baseline) + assert_includes out, "[snap_diff] 2 verified, 1 changed (1 re-recorded), 0 new (not verified)." + assert_includes out, "accepted it as the new baseline: changed" + assert_includes report, '"status":"rerecorded"' + assert_equal File.binread(fixture_image_path_from("b")), baseline + end + # Builds a throwaway git repo with COMMITTED baselines for `verified` and # `changed` (none for `new`), then runs the user's test file against it. # Returns the run's output, its exit status, and the HTML report it wrote # (nil when none was written). - def run_case(cases, reporter: true, record: nil) + def run_case(cases, reporter: true, record: nil, env: {}) Dir.mktmpdir do |dir| # macOS hands out /var/... symlinks; git reports the physical path, and # baseline lookup is a relative_path_from between the two. @@ -140,10 +175,12 @@ def run_case(cases, reporter: true, record: nil) Open3.capture2e(*git, "commit", "-qm", "baselines") out, status = Open3.capture2e( - {"SNAP_ROOT" => repo, "SNAP_IMAGES" => TEST_IMAGES_DIR.to_s, "SNAP_CASES" => cases, "CI" => nil, - "SNAP_NO_REPORTER" => (reporter ? nil : "1"), "SNAP_RECORD" => record}, + {"SNAP_ROOT" => repo, "SNAP_IMAGES" => TEST_IMAGES_DIR.to_s, "SNAP_CASES" => cases, + "CI" => nil, "SNAP_DIFF_ACCEPT_SCREENSHOTS" => nil, + "SNAP_NO_REPORTER" => (reporter ? nil : "1"), "SNAP_RECORD" => record}.merge(env), RbConfig.ruby, "-Ilib", "-Itest", file_fixture("summary_line_case.rb").to_s ) + yield repo if block_given? report = Pathname("#{repo}/screenshots/snap_diff_report.html") [out, status, (report.read if report.exist?)] end diff --git a/test/unit/record_modes_test.rb b/test/unit/record_modes_test.rb index 3311e2a9..f4950366 100644 --- a/test/unit/record_modes_test.rb +++ b/test/unit/record_modes_test.rb @@ -78,6 +78,58 @@ def after_teardown assert_equal :none, SnapDiff.config.record end + # --- SNAP_DIFF_ACCEPT_SCREENSHOTS -------------------------------------- + # + # The bulk-accept verb is a per-run decision, so it is an environment + # variable, not a line in test_helper.rb: the run's own intent outranks the + # permanent settings, which would otherwise decide every run. + + test "SNAP_DIFF_ACCEPT_SCREENSHOTS selects :all for this run, on CI or off it" do + SnapDiff.config.record = nil + + %w[1 true TRUE].each do |value| + with_env("SNAP_DIFF_ACCEPT_SCREENSHOTS", value) do + assert_equal :all, SnapDiff.config.record, value + with_ci("true") { assert_equal :all, SnapDiff.config.record, value } + end + end + end + + test "SNAP_DIFF_ACCEPT_SCREENSHOTS only counts when truthy" do + SnapDiff.config.record = nil + SnapDiff.config.fail_if_new = nil + + ["", "0", "false", "no"].each do |value| + with_env("SNAP_DIFF_ACCEPT_SCREENSHOTS", value) do + assert_equal :once, SnapDiff.config.record, value.inspect + with_ci("true") { assert_equal :none, SnapDiff.config.record, value.inspect } + end + end + end + + test "SNAP_DIFF_ACCEPT_SCREENSHOTS outranks an explicit record mode and fail_if_new" do + with_env("SNAP_DIFF_ACCEPT_SCREENSHOTS", "1") do + SnapDiff.config.record = :none + assert_equal :all, SnapDiff.config.record + + SnapDiff.config.record = nil + SnapDiff.config.fail_if_new = true + assert_equal :all, SnapDiff.config.record + end + end + + test "a per-screenshot record: outranks SNAP_DIFF_ACCEPT_SCREENSHOTS" do + SnapDiff.config.record = nil + + with_env("SNAP_DIFF_ACCEPT_SCREENSHOTS", "1") do + SnapDiff::Vcs.stub(:checkout_vcs, false) do + assert_raises(SnapDiff::ExpectationNotMet) do + SnapDiff::ScreenshotMatcher.new("c_#{Time.now.nsec}", record: :none).build_screenshot_assertion + end + end + end + end + # --- :once ------------------------------------------------------------ test ":once records a screenshot that has no committed baseline, and does not compare it" do @@ -372,7 +424,7 @@ def take_comparison_screenshot(_snapshot) # nothing and passes, with the "recorded" files thrown away with the box. # Refuse it under CI instead of building our own version of Percy. - test ":all refuses to run under CI" do + test "a :all written in code refuses to run under CI" do SnapDiff.config.record = :all error = with_ci("true") do @@ -385,6 +437,31 @@ def take_comparison_screenshot(_snapshot) assert_match(/CI/, error.message) end + test "a :all from SNAP_DIFF_ACCEPT_SCREENSHOTS runs under CI" do + SnapDiff.config.record = nil + + with_ci("true") do + with_env("SNAP_DIFF_ACCEPT_SCREENSHOTS", "1") do + SnapDiff::Vcs.stub(:checkout_vcs, true) do + snap = create_snapshot_for(:a, :c) + assertion = SnapDiff::ScreenshotMatcher.new(snap.full_name).build_screenshot_assertion + SnapDiff.session.verify + assert_equal :rerecorded, assertion.status + end + end + end + end + + test "a per-screenshot record: :all is refused under CI even with the variable falsy" do + with_ci("true") do + with_env("SNAP_DIFF_ACCEPT_SCREENSHOTS", "0") do + assert_raises(SnapDiff::ExpectationNotMet) do + SnapDiff::ScreenshotMatcher.new("c_#{Time.now.nsec}", record: :all).build_screenshot_assertion + end + end + end + end + test ":all runs off CI" do SnapDiff.config.record = :all @@ -549,18 +626,24 @@ def assert_not_captured(fixture, path, message) def setup super @original_ci = ENV.delete("CI") + @original_accept = ENV.delete("SNAP_DIFF_ACCEPT_SCREENSHOTS") end def teardown ENV["CI"] = @original_ci + ENV["SNAP_DIFF_ACCEPT_SCREENSHOTS"] = @original_accept super end - def with_ci(value) - original = ENV["CI"] - ENV["CI"] = value + def with_ci(value, &block) + with_env("CI", value, &block) + end + + def with_env(name, value) + original = ENV[name] + ENV[name] = value yield ensure - ENV["CI"] = original + ENV[name] = original end end