Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New Comparison Strategy using JSONassert #159

Open
connollyst opened this issue Jun 9, 2016 · 11 comments
Open

New Comparison Strategy using JSONassert #159

connollyst opened this issue Jun 9, 2016 · 11 comments

Comments

@connollyst
Copy link

I've written a new MongoComparisonStrategy based on the JSONassert project and really find it helps me debug my failing tests a lot quicker.

Background

For background, the default comparison strategy builds a query using the expected data as an example. So, if everything matches then great, but if any of the expected JSON doesn't match the actual object, the query fails. The only feedback you get is that the expected JSON wasn't found. This makes it very difficult to debug what the actual JSON was and what needs to be corrected.

How it Works

This new JsonComparisonStrategy queries each document by only it's _id field, and then compares the expected JSON to the actual JSON. The feedback is much more useful, for example:

com.lordofthejars.nosqlunit.core.NoSqlAssertionError: _id : xyz/two, authorization.admins
Expected: -rw
     got: --w

This tells you that the record with _id xyz/two for the subdocument field authorization.admins was expected to be -rw but was --w.

For comparison (har har) the default comparison strategy would report:

com.lordofthejars.nosqlunit.core.NoSqlAssertionError: Object # { "_id" : "xyz/two" , "project" : "xyz" , "name" : "two" , "displayName" : "Group Two" , "description" : "Group Two, of Namespace XYZ." , "order" : 2 , "retired" : true , "hidden" : true , "authorization" : { "users" : "-r-" , "admins" : "-rw"}} # is not found into collection [entity_groups]

My Questions

Two questions.

  • Should this be merged with a pull request? It includes a new dependency on org.skyscreamer.jsonassert so want to get input first.
    -- An alternative would be another maven module. Seems overkill but may be a good place for add-ons long term?
  • MongoDbAssertion has two top level functions public and the rest are private. I want to reuse a lot of these private functions, any objection to making them package-private instead?
@lordofthejars
Copy link
Owner

Absolutely. Thank you very much for your contribution, go ahead with the required changes.

Thanks.

@connollyst
Copy link
Author

Awesome, I'll be in touch with a PR.

@connollyst
Copy link
Author

Hey Alex,

I've got my implementation ready and am trying to write some unit tests. I was basing them off of MongoFlexibleComparisonStrategyTest to keep the code/test style consistent, but those tests are failing on my machine. Do they pass for you?

The error is

shouldThrowAnExceptionIfDifferentValuesInFlexibleStrategy(com.lordofthejars.nosqlunit.mongodb.MongoFlexibleComparisonStrategyTest)  Time elapsed: 0.04 sec  <<< FAILURE!
java.lang.AssertionError: Expected exception: com.lordofthejars.nosqlunit.core.NoSqlAssertionError
    at org.junit.internal.runners.statements.ExpectException.evaluate(ExpectException.java:32)
    at com.lordofthejars.nosqlunit.core.AbstractNoSqlTestRule$1.evaluate(AbstractNoSqlTestRule.java:72)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
    at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
    at org.junit.rules.RunRules.evaluate(RunRules.java:20)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
    at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
    at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray2(ReflectionUtils.java:208)
    at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:158)
    at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:86)
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:95)

The exception is declared as expected like so: @Test(expected=NoSqlAssertionError.class)
If I take out the Exception expectation part, then the test fails because the exception is thrown.

I tried removing that kind of Exception expectation and tried using the @Rule ExpectedException.. approach, no luck.

It almost seems like the @ShouldMatchDataSet is executed in some different scope than the @Test(expected=NoSqlAssertionError.class) Exception check. Just guessing though.

Any thoughts? If those tests work for you, might it be something about your environment setup?

Cheers,
Sean

@lordofthejars
Copy link
Owner

Currently I am on vacation. I will try next week.

Thanks

El 20 ago. 2016 6:58 p. m., "Sean Connolly" [email protected]
escribió:

Hey Alex,

I've got my implementation ready and am trying to write some unit tests. I
was basing them off of MongoFlexibleComparisonStrategyTest to keep the
code/test style consistent, but those tests are failing on my machine. Do
they pass for you?

The error is

shouldThrowAnExceptionIfDifferentValuesInFlexibleStrategy(com.lordofthejars.nosqlunit.mongodb.MongoFlexibleComparisonStrategyTest) Time elapsed: 0.04 sec <<< FAILURE!
java.lang.AssertionError: Expected exception: com.lordofthejars.nosqlunit.core.NoSqlAssertionError
at org.junit.internal.runners.statements.ExpectException.evaluate(ExpectException.java:32)
at com.lordofthejars.nosqlunit.core.AbstractNoSqlTestRule$1.evaluate(AbstractNoSqlTestRule.java:72)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray2(ReflectionUtils.java:208)
at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:158)
at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:86)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:95)

The exception is declared as expected like so: @test(expected=
NoSqlAssertionError.class)
If I take out the Exception expectation part, then the test fails because
the exception is thrown.

I tried removing that kind of Exception expectation and tried using the @rule
ExpectedException.. approach, no luck.

It almost seems like the @ShouldMatchDataSet is executed in some
different scope than the @test(expected=NoSqlAssertionError.class)
Exception check. Just guessing though.

Any thoughts? If those tests work for you, might it be something about
your environment setup?

Cheers,
Sean


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#159 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABcmYXD1yPh4tYyQrn7VsAuGF4eVvtNyks5qhzIdgaJpZM4IyfmH
.

@connollyst
Copy link
Author

Hi Alex,

Any update on this? I'd love to contribute this pull request if possible.

Cheers,
Sean

@lordofthejars
Copy link
Owner

You can contribute for sure, now I can release a new version pretty fast.

@connollyst
Copy link
Author

Great, good to hear.

Can you comment on the unit tests? When I check out the master branch and run the unit tests, many fail with the error I mentioned above. Want to make sure I'm starting with everything working correctly before I start mucking around.

Cheers,
Sean

@lordofthejars
Copy link
Owner

Test fixed, you can start working on this.

@connollyst
Copy link
Author

Hi Alex,

I still see a number of other failing tests. Are these failing on your side too?

-Sean

@lordofthejars
Copy link
Owner

lordofthejars commented Feb 10, 2017 via email

@connollyst
Copy link
Author

Sure, 42 of the 62 tests in nosqlunit-mongodb run, 5 fail. WhenMultipleMongoObjectsAreAnnotatedWithInject seems to run indefinitely and so the remaining tests never complete. The 5 failing tests are below.

WhenExpectedDataShouldBeCompared.less_expected_collection_than_database_collection_should_fail:

Expected :Expected collection names are [col1] but insert collection names are [col1, col3]
Actual   :Expected collection names are [col1] but insert collection names are [col3, col1]

WhenExpectedDataShouldBeCompared.expected_collection_item_has_less_attributes_than_database_collection_item_attribute_should_fail:

Expected :Expected DbObject and insert DbObject have different keys: Expected: [name] Inserted: [name, surname]
Actual   :Expected DbObject and insert DbObject have different keys: Expected: [name] Inserted: [surname, name]

WhenManagedMongoDbRuleIsRegistered.mongo_server_should_throw_an_exception_if_mongo_location_is_not_found:

Exception in thread "Thread-6" java.lang.IllegalStateException: Mongodb [/example--dbpathmongo-dbpath--port27017--logpathlogpath] could not be started. Next console message was thrown: Cannot run program "/example/bin/mongod" (in directory "target/mongo-temp"): error=2, No such file or directory
	at com.lordofthejars.nosqlunit.mongodb.ManagedMongoDbLifecycleManager$ProcessRunnable.prepareException(ManagedMongoDbLifecycleManager.java:354)
	at com.lordofthejars.nosqlunit.mongodb.ManagedMongoDbLifecycleManager$ProcessRunnable.run(ManagedMongoDbLifecycleManager.java:311)
	at java.lang.Thread.run(Thread.java:745)

WhenMongoDbRuleIsRegistered.should_clean_previous_data_and_insert_new_dataset_with_clean_insert_strategy:

com.lordofthejars.nosqlunit.core.NoSqlAssertionError: Expected collection names are [collection1, collection2] but insert collection names are [collection1, collection2, col1]

	at com.lordofthejars.nosqlunit.core.FailureHandler.createFailure(FailureHandler.java:7)
	at com.lordofthejars.nosqlunit.mongodb.MongoDbAssertion.checkCollectionsName(MongoDbAssertion.java:51)
	at com.lordofthejars.nosqlunit.mongodb.MongoDbAssertion.strictAssertEquals(MongoDbAssertion.java:32)
	at com.lordofthejars.nosqlunit.mongodb.DefaultComparisonStrategy.compare(DefaultComparisonStrategy.java:17)
	at com.lordofthejars.nosqlunit.mongodb.DefaultComparisonStrategy.compare(DefaultComparisonStrategy.java:10)
	at com.lordofthejars.nosqlunit.core.AbstractCustomizableDatabaseOperation.executeComparison(AbstractCustomizableDatabaseOperation.java:15)
	at com.lordofthejars.nosqlunit.mongodb.MongoOperation.compareData(MongoOperation.java:103)
	at com.lordofthejars.nosqlunit.mongodb.MongoOperation.databaseIs(MongoOperation.java:96)
	at com.lordofthejars.nosqlunit.core.AbstractNoSqlTestRule$1.assertExpectation(AbstractNoSqlTestRule.java:255)
	at com.lordofthejars.nosqlunit.core.AbstractNoSqlTestRule$1.evaluate(AbstractNoSqlTestRule.java:79)
	at com.lordofthejars.nosqlunit.mongodb.integration.WhenMongoDbRuleIsRegistered.should_clean_previous_data_and_insert_new_dataset_with_clean_insert_strategy(WhenMongoDbRuleIsRegistered.java:163)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
	at org.junit.runners.Suite.runChild(Suite.java:127)
	at org.junit.runners.Suite.runChild(Suite.java:26)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
	at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:237)
	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)

WhenMongoDbRuleIsRegistered.should_assert_if_expected_data_is_strict_equal:

com.lordofthejars.nosqlunit.core.NoSqlAssertionError: Expected collection names are [collection1, collection2] but insert collection names are [collection1, collection2, col1]

	at com.lordofthejars.nosqlunit.core.FailureHandler.createFailure(FailureHandler.java:7)
	at com.lordofthejars.nosqlunit.mongodb.MongoDbAssertion.checkCollectionsName(MongoDbAssertion.java:51)
	at com.lordofthejars.nosqlunit.mongodb.MongoDbAssertion.strictAssertEquals(MongoDbAssertion.java:32)
	at com.lordofthejars.nosqlunit.mongodb.DefaultComparisonStrategy.compare(DefaultComparisonStrategy.java:17)
	at com.lordofthejars.nosqlunit.mongodb.DefaultComparisonStrategy.compare(DefaultComparisonStrategy.java:10)
	at com.lordofthejars.nosqlunit.core.AbstractCustomizableDatabaseOperation.executeComparison(AbstractCustomizableDatabaseOperation.java:15)
	at com.lordofthejars.nosqlunit.mongodb.MongoOperation.compareData(MongoOperation.java:103)
	at com.lordofthejars.nosqlunit.mongodb.MongoOperation.databaseIs(MongoOperation.java:96)
	at com.lordofthejars.nosqlunit.core.AbstractNoSqlTestRule$1.assertExpectation(AbstractNoSqlTestRule.java:255)
	at com.lordofthejars.nosqlunit.core.AbstractNoSqlTestRule$1.evaluate(AbstractNoSqlTestRule.java:79)
	at com.lordofthejars.nosqlunit.mongodb.integration.WhenMongoDbRuleIsRegistered.should_assert_if_expected_data_is_strict_equal(WhenMongoDbRuleIsRegistered.java:82)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
	at org.junit.runners.Suite.runChild(Suite.java:127)
	at org.junit.runners.Suite.runChild(Suite.java:26)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
	at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:237)
	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)

I want to get all tests running & passing before I make any code changes so I know I don't break anything by adding this new feature.

Cheers,
Sean

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants