it 'does not send notification email when user is invalid' do
expect(NotificationMailer).not_to receive(:user_notification)
post :create, user: attributes_for(:invalid_user)
end
So, what this is doing is set you're expectation just the way you did, and then post to the user_controller create method the invalid_user attributes.
Of course, the post shouldn't be allowed to create the record if you have set your validations correctly in the user model, and subsequently not call NotificationMailer.user_notification.
Note that attributes_for is another FactoryGirl method that you can use to arrange and pass your factory attributes as controller params.
Now! why does it not work with your original approach?
It is is because FactoryGirl is complaining that it cannot create the record, and that is absolutely logical since you're trying to create an invalid user. The failing error has nothing to do with testing your email notification, but rather with the way you setup your Factory.
Final note! If you run the test and it complains:
"NoMethodError: undefined method `post' for #<RSpec::ExampleGroups"
It probably means that your spec file is not located under spec/controllers.
post, create, patch, delete methods are part of the RSpec::Rails::ControllerExampleGroup
To solve this please refer to the following Stackoverflow answer
Hope this helps.