-
-
Notifications
You must be signed in to change notification settings - Fork 102
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix diff output when a fuzzy finder anything is inside an expected hash #599
Fix diff output when a fuzzy finder anything is inside an expected hash #599
Conversation
in 1.8.7 Looks great otherwise, thank you! |
For the elevator pitch, may I kindly as you to add before and after example outputs to the pr description? |
daa5bb5
to
4c22e70
Compare
Ready!
|
lib/rspec/support/differ.rb
Outdated
@@ -73,6 +84,14 @@ def initialize(opts={}) | |||
|
|||
private | |||
|
|||
def hash_with_anything?(arg) | |||
Hash === arg && safely_flatten(arg).any? { |a| RSpec::Mocks::ArgumentMatchers::AnyArgMatcher === a } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It seems this is due to this. We call safely_flatten with a hash arg, and it calls flatten on it.
This will break 1.8.7. We can’t afford this. Even though I’m a proponent of soft-deprecating older Rubies in our code, this doesn’t include breaking hypothetical suites that might still exist.
Can the same be achieved somehow differently? Like taking .values
from the hash?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please disregard my note regarding breaking 1.8.7, I’ve just noticed that you have a check for this.
Still, does it make sense to flatten? We could just check the too level of the hash, right?
lib/rspec/support/differ.rb
Outdated
Hash === arg && safely_flatten(arg).any? { |a| RSpec::Mocks::ArgumentMatchers::AnyArgMatcher === a } | ||
end | ||
|
||
def no_procs_and_no_numbers?(*args) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I’m also struggling to understand how this works.
We do pass expected and actual there, and they end up wrapped in an array when flatten is called on it?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Indeed... that's weird... What if I remove the star operator from #no_procs
and no_numbers
method?
The purpose of this no_procs_and_no_numbers?
method is to reduce the number of &&
operators used in the #diff
method, thus appeasing rubocop by removing the Metrics::PerceivedComplexity
offense.
I've just simplified I don't know if that's the best solution, the downside is it modifies the original Other solutions I could have taken that didn't need to alter the
def no_procs_and_no_numbers?(*args)
safely_flatten(args).none? { |a| Proc === a } && safely_flatten(args).none? { |a| Numeric === a }
end
def no_procs_and_no_numbers?(*args)
safely_flatten(args).none? { |a| Proc === a || Numeric === a }
end Both alternatives makes all test pass on my local machine. |
lib/rspec/support/differ.rb
Outdated
end | ||
|
||
def no_procs_and_no_numbers?(*args) | ||
no_procs?(args) && no_numbers?(args) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
args here will be an array of two elements?
How does it work to flatten an array of two hashes?
Can we check no_procs?(expected) && no_procs?(actual) && no_numbers? ….
Each if those methods runs the (potentially expensive?) safe_flatten. I’d prefer this to be done just once (or, better - never!)
Can’t we just take top-level keys of each of those arrays without flattening recursively?
This would work on 1.8.7
No tricks with has/array structures
Early exit if there is a proc/number
No extra checks in nested values that we don’t care about
Less code
No splatting/desplatting
UPDATE: yeah... what you are mentioning here is a good idea: get rid of the |
I realized the change is simpler than I thought. I just introduced a new commit that reverts the I am doing this instead of using
def no_procs?(*args)
args.flatten.none? { |a| Proc === a }
end But that method was only called once in that old file. You can check it out with this command:
This leads me to think that before using def no_procs?(actual, expected)
(Proc != actual) && (Proc != expected)
end The fancy way is more readable, and it has the benefit if But again, the code changes I'm proposing are narrowed down to a specific case scenario, where the user wants to diff two hashes in which one of them an |
77aa475
to
e5de130
Compare
e5de130
to
c3e9ea9
Compare
237db12
to
416cd49
Compare
I've just submitted another commit, because I noticed another issue: on this line, the expected var mutates. So, if you provide Hopefully the spec I've added will explain what the issue better than how I am explaining it here. Here is the code snippet: it "checks the 'expected' var continues having the 'anything' fuzzy matcher, it has not mutated" do
actual = { :fixed => "fixed", :trigger => "trigger", :anything_key => "bcdd0399-1cfe-4de1-a481-ca6b17d41ed8" }
expected = { :fixed => "fixed", :trigger => "wrong", :anything_key => anything }
differ.diff(actual, expected)
expect(expected).to eq({ :fixed => "fixed", :trigger => "wrong", :anything_key => anything })
end Here is the output before the application code changes (failing test)
Here is the output AFTER the application code changes (test succeeds) |
Nice find. Can this be the root cause of this getting its way to the diff in the first place?
|
lib/rspec/support/differ.rb
Outdated
@@ -56,6 +58,28 @@ def diff_as_string(actual, expected) | |||
end | |||
# rubocop:enable Metrics/MethodLength | |||
|
|||
def diff_hashes_as_object(actual, expected) | |||
if defined?(RSpec::Mocks::ArgumentMatchers::AnyArgMatcher) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We only call this from all_hashes?, and it already has this check
Potentially, this method can be mistakenly called from somewhere else causing errors for those not using rspec-mocks, but i’m less worried about this at this point.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, it is currently called only under if all_hashes?(actual, expected)
condition, and all_hashes?
is also checking if the user is opting out rspec-mocks
. This check in line 62 is redundant if we assume this method will only be called inside all_hashes?
condition . But it is there in case someone mistakenly call the method from somewhere else, so the program won't break if rspec-mocks
is opted-out.
If someone opts-out rspec-mock
, diff_hashes_as_object
should have the exact same behavior as diff_as_object
. That's the purpose of this redundant check.
Personally, I like redundancy. But I understand sometimes it may be overkill or unnecessary... maybe defining diff_hashes_as_object
as private is good enough? I don't have a strong opinion here.
@@ -77,6 +101,10 @@ def no_procs?(*args) | |||
safely_flatten(args).none? { |a| Proc === a } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What I really liked about your previous changes was the potential to get rid of the recursive check and the safely_flatten method.
No pressure, as this is not the goal of the pr, but it would be a nice bonus to fixing the issue. And a performance improvement in theory.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I understand, it would be nice to get rid of that recursivity. You can count on me to brainstorm on another PR or submit a proposal to fix that issue. I am enjoying reading and getting my hands on this project, and once this PR is finished (by finished I mean either merged or closed (if you and the other maintainers believe the changes I'm proposing here are too risky, or not worth doing them)) I'd like to propose some changes to make the BuiltIn::Change
matcher class diffable when it tries to match a Hash in rspec-expectations
repo... but that's another story.
About your concern with safely_flatten
, that method was introduced ~10 years ago. Does anyone still knows why that method is there? I think its purpose is not only to check in a one method call whether all the arguments are not Proc
, but along with no_numbers?
to make Differ
work with deep down nested arrays inside actual
and expected
vars. If we stick with my previous change, we can make the method faster, but we may loose the Differ
feature to diff deep down nested arrays in a seamlessly way.
Just some brainstorming and ramble here: Is it worth to keep that recursivity? when an expected
and actual
vars with deep nested arrays are submitted as input to Differ
, is worth to display a big diff output? are big diff strings useful when you have to scroll down pages of information? or a diff string is useful only when it shows the exact difference and a little bit of context? Maybe the answers to these questions will pop up when we will see such diff strings... I don't know the answer.
On the project I am currently working on, when I'm testing the actual and expected values of a hash with nested hashes inside, I've found is useful to strip my actual hash of its nested objects in my test set-up, and inside an aggregate_failures
block use one expect(...)...
sentence for the main hash, and for each nested object use an extra expect(...)...
sentence. That's a nice workaround instead of testing a big hash in a single expect(...)...
sentence and expect a single diff to show me -everything- I want to see (the idea behind #596 )
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I can’t tell of the top of my head with certainty, but I can’t recall if nested hashes are (usefully) diffed.
I’d rather simplify the code if no spec fails.
spec/rspec/support/differ_spec.rb
Outdated
expected_diff = dedent(<<-'EOD') | ||
| | ||
|@@ -1,4 +1,4 @@ | ||
| :anything_key => "bcdd0399-1cfe-4de1-a481-ca6b17d41ed8", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Now when I look at this, this is indeed and improvement to -anything_key +anything_key
we had.
But do we need a potentially huge value here? An uuid is fine, but what about a 5kb json-like thing?
Most certainly if we’ve anything
ed it in the test, we don’t care what’s inside, including the diff?
@JonRowe does this make sense?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are you concerned by the case a 5kb json-like thing would be copied into the expected -> anything
value? that will definitively be a performance issue. Would it be worth doing the change? Is there a way we can make sure the 5kb json-like thing will be a shallow copy? Maybe a shallow copy will mitigate the performance issue.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe a shallow copy will not impact memory ram consumption. But what about CPU time? does diff_as_object
iterates the actual
and expected
vars regardless of their .object_id
? or it checks the object_id
and if they match it will cease any iteration?
Another question: rspec-support
module should be allowed to mutate expected
vars? regardless they will be later on return them to their original state? My proposed changes on this PR are a hacky way to fold noise when the developers use anything
fuzzy matcher on their hash, I understand this may not be allowed to do here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Apparently, a 5kb json-like thing will impact CPU time ONLY if the matcher fails.
def diff_as_object(actual, expected)
actual_as_string = object_to_string(actual) # STEP 1
expected_as_string = object_to_string(expected) # STEP 2
diff_as_string(actual_as_string, expected_as_string) # STEP 3
end
STEP 1 and STEP 2 will get the string representation of 5kb json-like thing in two different strings. STEP 3 will perform the comparison of both strings. This will be an expensive operation because both strings will be huge. In contrast, without the changes on this PR, comparing the string representation of this 5kb json-like thing with anything
fuzzy matcher would have been easy-peasy
AFAIK, RSpec::Support::Differ
is called when an expectation matcher fails AND the expectation matcher responds to diffable?. If the matcher does not responds to diffable, expected
and actual
vars are assigned nil
and RSpec::Support::Differ
will not perform any action at all.
My assessment is this will impact performance only when the test fails and the developer is using a matcher that produces an output-diff. But this is my assessment about performance, I still don't know the answer to: should rspec-support
module be allowed to mutate expected
vars in the developer test bench (ie, his _spec.rb
files)? or rspec-support
should remain a "pure" read-only module?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think @pirj's concern was about larger diffs, but as things should be identical the differ will often not print anything at all
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Identical - yes. But if it’s neighbouring with the difference, like the ‘anything_key’ here on line 567, and it is big, i would prefer to see the literal “anything” to be printed rather than 5kb of json.
Yes, the cause of:
is the mutation there: First I take the Before submitting the latest commit c3e9ea926463829a048a131f7b85fef16c186ef5 , I thought the mutation would occur inside the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry for the delayed review, I think this localised change makes sense as it should reduce diff confusion. I have some suggestions though.
lib/rspec/support/differ.rb
Outdated
def all_hashes?(actual, expected) | ||
defined?(RSpec::Mocks::ArgumentMatchers::AnyArgMatcher) && (Hash === actual) && (Hash === expected) | ||
end |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wouldn't check if we have an any arg here, the other method already does this.
def all_hashes?(actual, expected) | |
defined?(RSpec::Mocks::ArgumentMatchers::AnyArgMatcher) && (Hash === actual) && (Hash === expected) | |
end | |
def all_hashes?(actual, expected) | |
(Hash === actual) && (Hash === expected) | |
end |
lib/rspec/support/differ.rb
Outdated
def diff_hashes_as_object(actual, expected) | ||
if defined?(RSpec::Mocks::ArgumentMatchers::AnyArgMatcher) | ||
anything_hash = expected.select { |_, v| RSpec::Mocks::ArgumentMatchers::AnyArgMatcher === v } | ||
|
||
anything_hash.each_key do |k| | ||
expected[k] = actual[k] | ||
end | ||
|
||
diff_string = diff_as_object(actual, expected) | ||
|
||
anything_hash.each do |k, v| | ||
expected[k] = v | ||
end | ||
|
||
diff_string | ||
else | ||
diff_as_object(actual, expected) | ||
end | ||
end |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Typically we define such methods conditionally rather than doing checks in the method e.g.
def diff_hashes_as_object(actual, expected) | |
if defined?(RSpec::Mocks::ArgumentMatchers::AnyArgMatcher) | |
anything_hash = expected.select { |_, v| RSpec::Mocks::ArgumentMatchers::AnyArgMatcher === v } | |
anything_hash.each_key do |k| | |
expected[k] = actual[k] | |
end | |
diff_string = diff_as_object(actual, expected) | |
anything_hash.each do |k, v| | |
expected[k] = v | |
end | |
diff_string | |
else | |
diff_as_object(actual, expected) | |
end | |
end | |
if defined?(RSpec::Mocks::ArgumentMatchers::AnyArgMatcher) | |
def diff_hashes_as_object(actual, expected) | |
anything_hash = expected.select { |_, v| RSpec::Mocks::ArgumentMatchers::AnyArgMatcher === v } | |
anything_hash.each_key do |k| | |
expected[k] = actual[k] | |
end | |
diff_string = diff_as_object(actual, expected) | |
anything_hash.each do |k, v| | |
expected[k] = v | |
end | |
diff_string | |
end | |
else | |
def diff_hashes_as_object(actual, expected) | |
diff_as_object(actual, expected) | |
end | |
end |
I'd also like to see a hash built from expected rather than mutating the original hash e.g.
expected_to_diff =
expected.reduce({}) do |hash, (key, value)|
if RSpec::Mocks::ArgumentMatchers::AnyArgMatcher === v
hash[key] = actual[key]
else
hash[key] = expected[key]
end
hash
end
This has the benefit of less enumerations as well as a safety aspect
spec/rspec/support/differ_spec.rb
Outdated
expected_diff = dedent(<<-'EOD') | ||
| | ||
|@@ -1,4 +1,4 @@ | ||
| :anything_key => "bcdd0399-1cfe-4de1-a481-ca6b17d41ed8", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think @pirj's concern was about larger diffs, but as things should be identical the differ will often not print anything at all
15843e3
to
dd7a4c7
Compare
Almost ready @JonRowe .
If I run only that test file with the command: bundle exec rspec ./spec/rspec/support/differ_spec.rb A failure rises. Because the RSpec.describe Differ sentence But if you run the full test suite with: bundle exec rspec spec Chances are all tests will pass, unless I don't know how to fix this issue, is there a way to force the test suite to not run I've tried to dig in and find where What should we do about this? |
What if we quote it? RSpec.describe "Differ" do it’s the way we recommend writing specs anyway. Would it solve the flakiness issue? Alternatively. We autoload classes, and numerous places may trigger |
That's awesome! If there are no more changes left to do, I can squash all the commits into a single one. BTW, What's the difference between RSpec.describe the class and the name of the class as string? |
Nice! Let me have another quick look.
This autoloading (this is the first in my practice). And also probably |
on 2.7 🤔 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are a few things with potential improvements in this code. But I don’t think they are related to this PR.
The change looks good. Thank you!
You're welcome! Is there anything else I can do for you here? @pirj @JonRowe ? I can squash all the commits into a single one if you think this PR is OK. BTW, in Ruby 2.7 build it is complaining about spec/rspec/support/differ_spec.rb:169
I don't know how can I reproduce this error on my local machine. I have not touched this line 169 before, is it a flaky test? or is it related to changing: RSpec.describe Differ do to RSpec.describe "Differ" do ? |
2.7 passed on re-run. Must be order-dependent load order flakiness. Let’s see if it bugs us again. But I don’t think it should stop from merging. Thanks again. |
Ugh I hate to say this at this late juncture, but I think @pirj's earlier point about large nested hashes has a point, if our point here is to make the diff smaller by making things equal, and the matcher is matching on |
That's thinking outside of the box! I tried your suggestion and it works! I didn't thought about it... but using the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just perfect now 🙌 Thank you!
And thanks @JonRowe for the smart suggestion.
expected_diff = dedent(<<-'EOD') | ||
| | ||
|@@ -1,4 +1,4 @@ | ||
| :anything_key => anything, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🤩
Co-authored-by: Jon Rowe <mail@jonrowe.co.uk>
HunkGenerator again, flaky |
Thanks! |
This is a simplified version of PR #596 , it solves the issue #551 "Diff reports confusing output when used with "fuzzy" matchers like
anything
"This PR will fix only
anything
values associated to top level keys of a hash. It will not work with nested hashes.Example output BEFORE changes in this PR:
Example output AFTER changes in this PR: