Don't Use Mocks (Or Do).

Tim Malseed -

Disambiguating Test Language

Testing in software engineering is not just about writing tests, it's about building a testing strategy that fits your codebase and your team's needs. The challenge lies in understanding what to test, how to test it, and how to make sure tests are effective, maintainable and meaningful. Tests should be easy to refactor, and robust.

Testing strategies vary significantly depending on the platform, tools and team maturity. What works well in backend services might not work as effectively on Android. These differences can lead to confusion and debates about the 'right' way to test.

One phrase I've been hearing for years now is 'don't use mocks'. And, like a lot of tech dogmatism, I find it really unhelpful. Sometimes I use mocks, and sometimes I don't. Sometimes they're helpful, and sometimes they aren't.

One key to solidifying your approach to testing is understanding the purpose and proper usage of test doubles. These are objects used in tests to stand in for actual components, allowing you to isolate and test specific behaviour in your code. The terminology around test doubles - such as dummy, stub, fake, and mock - is often used interchangeably, and can lead to misunderstandings.

This post aims to clear up the confusion by disambiguating these terms.

Disambiguation

Let's start by understanding the terminology around unit testing

Test Double:
Any type of object that can stand in for the real component during testing. The following are all types of test doubles:

  • Dummy
    A placeholder object that isn't actually used for the test
  • Stub
    Provides a predefined response to function calls
  • Fake
    Mimics the behaviour of a component, but with simplified logic
  • Mock
    Used to verify interactions between a class and its dependencies

Examples

Suppose we have the following UserManager class, which allows us to set and get a userName from some storage interface.

interface Storage {  
    fun save(key: String, value: String)  
    fun get(key: String): String?  
}  
  
class UserManager(private val storage: Storage) {  
  
    fun saveUsername(username: String) {  
        storage.save("username_key", userName)  
    }  
  
    fun getUsername(): String? {  
        return storage.get("username_key")  
    }  
}

Suppose that the default implementation of Storage is some concrete class, let's say SharedPreferencesStorage:

class SharedPreferencesStorage(val preferences: SharedPreferences) : Storage {

	override fun save(key: String, value: String) {
		preferences.edit().putString(key, value)
	}

	...

}

To test the UserManager we create an instance, set a userName, and then get the userName and assert that it matches the value we set:

class UserManagerTest {  
  
    @Test  
    fun test() {  
	    // Arrange
        val userManager = UserManager(storage = ?? )  

		// Act
        userManager.saveUsername(username = "my_username") 

		// Assert
        assertEquals("my_username", userManager.getUsername())  
    }  
}

We need an instance of Storage in order to test the UserManager. But, we can't use theSharedPreferencesStorage class, as it requires a SharedPreferences.  And besides, we don't have much control over the SharedPreferenceStorage class. We want to change how it behaves so that we can exercise our UserManager class.

Test Doubles

What we need here is a 'test double' - something that can stand in for the Storage  interface in our UserManager.

Dummy

Reminder: A dummy is a placeholder object that is passed around, but isn't actually used in the test.

class UserManagerTest {  

	val dummyStorage = object: Storage {
	
		override fun save(key: String, value: String) {
			// Do nothing
		}
		
	    override fun get(key: String): String? {
		    return null
	    }
	}

    @Test  
    fun test() {
	    // Arrange
        val userManager = UserManager(storage = dummyStorage)  
  
        // Nothing to assert
    }  
}

Here, we've created a dummy implementation of Storage, which doesn't do anything useful. It just exists so we have something to pass to the UserManager constructor. We don't care about how it behaves under test.

Note: In this example, we're really limited in what sort of assertions we can perform. Since UserManager heavily relies on Storage, and our dummy Storage implementation doesn't do anything, there isn't really anything for us to test. In the real world, a test dummy might be used to satisfy some minor dependency used for side effects or unimportant code, whose behaviour doesn't influence the thing you're trying to test.

Stub

Reminder: A stub is a test double that provides predefined responses to function calls.

Stubs don't simulate real behaviour, they just control the inputs or outputs of the thing we're testing.

class StubStorage : Storage {  
    override fun save(key: String, value: String) {  
        // Do nothing
    }  
  
    override fun get(key: String): String? {  
        return if (key == "username_key") "StubUser" else null  
    }  
}


class UserManagerTest {  

    @Test  
    fun test() {  
	    // Arrange
        val userManager = UserManager(storage = StubStorage())  

		// Assert
        assertEquals("StubUser", userManager.getUsername())  
    }  
}

Like dummies, stubs limit the value of the tests you can perform. getUsername() is always going to return "StubUser", regardless of what you pass to saveUsername(). The test above really just tells us that userManager.getUsername() calls storage.get().

Fake

Reminder: A fake is a test double that mimics the behaviour of a real component, but with simplified logic.

class FakeStorage : Storage {  
    private val data = mutableMapOf<String, String>()  
  
    override fun save(key: String, value: String) {  
        data[key] = value  
    }  
  
    override fun get(key: String): String? {  
        return data[key]  
    }  
}

class UserManagerTest {  

    @Test  
    fun test() {  
	    // Arrange
        val userManager = UserManager(storage = FakeStorage())  

		// Act 
		userManager.saveUsername("TestUser") 

		// Assert
        assertEquals("TestUser", userManager.getUsername())  
    }  
}

Because the fake functions in a similar fashion to the real, concrete Storage implementation that our production code uses, we're able to make assertions about sequences of events in the UserManager. In this example, we can check that after saving the username, attempting to retrieve the username will return the same value.

Mock

Reminder: A mock is used to verify that specific interactions occurred between a class and its dependencies. Below is an example of a 'manual mock' - but typically we'd use a mocking framework like Mockito or Mockk. More on that later on.

class MockStorage : Storage {  
    var saveCalled = false  
    var savedKey: String? = null  
    var savedValue: String? = null  
  
    override fun save(key: String, value: String) {  
        saveCalled = true  
        savedKey = key  
        savedValue = value  
    }  
  
    override fun get(key: String): String? {  
        return null
    }  
}

class UserManagerTest {  
  
    @Test  
    fun test() {  
        // Arrange  
        val mockStorage = MockStorage()  
        val userManager = UserManager(storage = mockStorage)  
  
        // Act  
        userManager.saveUsername("TestUser")  
  
        // Assert  
        assert(mockStorage.saveCalled) 
        assert(mockStorage.savedKey == "username_key") 
        assert(mockStorage.savedValue == "TestUser") 
    }  
}

Testing Strategies

Deciding which test double makes the most sense for your test depends on the type of test you're writing.

Two common strategies for testing code are state tests and behavioural tests.

State Testing

State testing focuses on what the system under test returns, or how it changes internally after an action. The goal is to ensure that the system produces the expected output, or reflects the expected state, after some operations, without concern for its dependencies.

State testing relies on Fakes and Stubs.

State testing is generally preferred for testing business logic.

  1. Focus on observable behaviour
    State testing examines what the system produces, not how it got there. This makes it more aligned with the user's or product's expectations, ensuring that the system's public API behaves correctly.
  2. Reduced fragility
    Tests that focus on state are more resistant to changes in the implementation details. If you refactor some internal logic, the test doesn't need to change as long as the public behaviour remains the same
  3. Better encapsulation
    State testing ensures that you only test through public APIs, which promotes better encapsulation. When tests rely on internal methods, it exposes internal details that could change frequently. By focusing on states, you ensure that you are only testing how the system should behave from an external perspective.

Behavioural Testing

Behavioural testing focuses on how the system interacts with its dependencies, ensuring that the correct methods are called with the correct parameters.

Behavioural testing relies on Mocks.

Behavioural testing is necessary when dealing with external dependencies, such as databases, networking, logging or analytics tracking. The interactions themselves are what you care about in these cases.

  1. Validating side effects
    Behavioural testing is about ensuring that your code interacts correctly with external systems. These external dependencies often have side effects (e.g. writing data to a database, sending a network request), and you need to confirm that the expected interactions occur.
  2. Testing critical external dependencies
    External systems often require specific protocols to be followed (e.g. saving data to a database or sending metrics to an analytics service). Behavioural testing ensures that the correct calls are made, in the right order, with the correct arguments.

Which strategy should I use?

It depends! Use state testing for your core/business logic, and use behavioural testing to test interactions with external dependencies. A good testing strategy will include a mixture of both,  with a strong bias towards state testing.

Here's an example that uses both state and behavioural testing.

Note: This test uses a combination of fake and mock.

class MyViewModel(
    private val repository: UserRepository,
    private val analytics: AnalyticsTracker
) : ViewModel() {

    var someState: String = "InitialState"

    fun performAction() {
        val user = repository.getUser()  
        someState = "ActionCompleted"
        analytics.trackEvent("ActionPerformed", mapOf("user" to user.name))
    }
}


class MyViewModelTest {

    @Test
    fun `test performAction updates state and calls analytics`() {
        // Arrange
        val fakeRepo = FakeUserRepository(listOf(User("TestUser")))
        val mockAnalytics = mock(AnalyticsTracker::class.java)
        val viewModel = MyViewModel(fakeRepo, mockAnalytics)

        // Act
        viewModel.performAction()

        // Assert (State Testing)
        assertEquals("ActionCompleted", viewModel.someState)

        // Assert (Behavioural Testing)
        verify(mockAnalytics).trackEvent(
            eq("ActionPerformed"), 
            eq(mapOf("user" to "TestUser"))
        )
    }
}

The above example is actually a very common test that I've written for Android apps.

Mainly, I want to confirm that the state correctly changes after the user performs an action - to ensure correctness of the business logic. But, I also want to make sure that analytics were correctly sent off to the analytics service.

If someone refactors performAction() and they accidentally remove the trackEvent call, the test will remind them.

What about frameworks like Mockito or Mockk

Mocking frameworks can add to the confusion somewhat, as they tend to offer lots of different features, and not just mocks. Their main advantage is to facilitate behavioural testing, however they can be used to help generate stubs and fakes for state based testing as well.

For example, MockK includes functionality to automatically generate test doubles. But, are these dummy, stub, fake or mock?

It actually depends on how the test double is configured and used.

Dummy:

val dummyLogger = mockk<Logger>()
val viewModel = MyViewModel(dummyLogger)

If our test doesn't care about how the logger behaved, then dummyLogger effectively acts as a dummy.

Stub:

val stubRepo = mockk<UserRepository>()
every { stubRepo.getUser() } returns User("StubUser")

Here, our mock just returns a predefined value whenever getUser() is called - our mock is acting as a stub.

Fake:

It's possible to have a mock simulate a fake in some scenarios. This is done by a combination of stubbing and internal state:


val mockPreferences = mockk<SharedPreferences>(relaxed = true)
val mockEditor = mockk<SharedPreferences.Editor>(relaxed = true)

val fakeStorage = mutableMapOf<String, String>() // internal state

every { mockPreferences.getString(any(), any()) } answers { 
    fakeStorage[firstArg()] ?: secondArg()
}

every { mockEditor.putString(any(), any()) } answers {
    val key = firstArg<String>()
    val value = secondArg<String>()
    fakeStorage[key] = value
    mockEditor  
}


@Test
fun `test saving and retrieving from SharedPreferences`() {
    
    mockEditor.putString("username", "TestUser").apply()

    val result = mockPreferences.getString("username", "DefaultUser")

    assertEquals("TestUser", result)
}

Mock:

val mockLogger = mockk<Logger>()
val viewModel = MyViewModel(mockLogger)

// Act
viewModel.performAction()

// Verify interaction with mock
verify { mockLogger.log("Action performed") }

Mockito and Mockk mocks keep track of which functions were called, and how many times. The verify function offered by Mockk is used to check that the log() function was called.

Should I use mocking frameworks?

Mocking frameworks can help reduce boilerplate and provide a convenient way to perform behaviour testing. Since mocking framework mocks can act as stubs or fakes, they can also assist with state based testing.

However, due to their ease of use and quick setup, mocking frameworks can encourage the proliferation of behaviour tests where state tests might be more appropriate. Remember, state tests are generally preferred when testing business logic, and behaviour tests are more appropriate for testing interactions with external dependencies.

Codebase and team maturity

No codebase is perfect, and not every team knows how to write testable code.

Some codebases pass concrete dependencies around, or don't leverage dependency injection, which makes it harder to swap dependencies out for other test doubles.

Some codebases rely heavily on calling external services, and in the correct order, and the correct number of times.

Other codebases are so loosely coupled that there are more and more layers to test, and each test seems really low value.

There's no right answer when it comes to testing. Whether you use state based testing, behavioural testing, dummies, stubs, fakes or mocks - really depends on what you're trying to achieve.

Use mocks when you're trying to test behaviour, and use fakes and stubs when you're testing state. Use a mocking framework if it saves you time and reduces boilerplate. Do whatever works for your codebase and your team. Don't let dogmatism stop you from using the tools at your disposal.