Roborazzi

Make JVM Android Integration Test Visible

Roborazzi now supports Robolectric Native Graphics (RNG) and enables screenshot testing.📣

Why Choose Roborazzi?

Why is screenshot testing important?

Screenshot testing is key to validate your app's appearance and functionality. It efficiently detects visual issues and tests the app as users would use it, making it easier to spot problems. It's quicker than writing many assert statements, ensuring your app looks right and behaves correctly.

What are JVM tests and why test with JVM instead of on Android?

JVM tests, also known as local tests, are placed in the test/ directory and are run on a developer's PC or CI environment. On the other hand, device tests, also known as Instrumentation tests, are written in the androidTest/ directory and are run on real devices or emulators. Device testing can result in frequent failures due to the device environment, leading to false negatives. These failures are often hard to reproduce, making them tough to resolve.

Paparazzi and Roborazzi: A Comparison

Paparazzi is a great tool for visualizing displays within the JVM. However, it's incompatible with Robolectric, which also mocks the Android framework.

Roborazzi fills this gap. It integrates with Robolectric, allowing tests to run with Hilt and interact with components. Essentially, Roborazzi enhances Paparazzi's capabilities, providing a more efficient and reliable testing process by capturing screenshots with Robolectric.

Leveraging Roborazzi in Test Architecture: An Example

Integrating Roborazzi into the Architecture: An Example from DroidKaigi 2023 App

In the DroidKaigi 2023 app, Roborazzi was introduced from the early stages of development as part of the architectural design. This integration allowed the team to verify changes throughout the development process. The specific architectural decisions and how they were implemented can be found README.

Try it out

Available on Maven Central.

Add Robolectric

This is an example of adding Robolectric to your project: https://github.com/takahirom/roborazzi-usage-examples/compare/b697...5c12

This library is dependent on Robolectric. Please see below to add Robolectric.

https://robolectric.org/getting-started/

Add Roborazzi

This is an example of adding Roborazzi to your project: https://github.com/takahirom/roborazzi-usage-examples/commit/3a02

To take screenshots, please use Robolectric 4.10 alpha 1 or later and please add @GraphicsMode(GraphicsMode.Mode.NATIVE) to your test class.

@GraphicsMode(GraphicsMode.Mode.NATIVE)

Build setup

Roborazzi is available on maven central.

This plugin simply creates Gradle tasks record, verify, compare and passes the configuration to the test.

build.gradle.kts

plugins buildscript

Define plugin in root build.gradle.kts

plugins {
  ...
  id("io.github.takahirom.roborazzi") version "[version]" apply false
}

Apply plugin in module build.gradle.kts

plugins {
  ...
  id("io.github.takahirom.roborazzi")
}

root build.gradle.kts

buildscript {
  dependencies {
    ...
    classpath("io.github.takahirom.roborazzi:roborazzi-gradle-plugin:[version]")
  }
}

module build.gradle.kts

plugins {
    ...
    id("io.github.takahirom.roborazzi")
}

build.gradle version

plugins buildscript

Define plugin in root build.gradle

plugins {
  ...
  id "io.github.takahirom.roborazzi" version "[version]" apply false
}

Apply plugin in module build.gradle

plugins {
  ...
  id 'io.github.takahirom.roborazzi'
}

root build.gradle

buildscript {
  dependencies {
    ...
    classpath "io.github.takahirom.roborazzi:roborazzi-gradle-plugin:[version]"
  }
}

module build.gradle

apply plugin: "io.github.takahirom.roborazzi"

Use Roborazzi task	Use default unit test task	Description
`./gradlew recordRoborazziDebug`	`./gradlew testDebugUnitTest` after adding `roborazzi.test.record=true` to your gradle.properties file. or `./gradlew testDebugUnitTest -Proborazzi.test.record=true`	Record a screenshot Default output directory is `build/outputs/roborazzi` You can check a report under `build/reports/roborazzi/index.html`
`./gradlew compareRoborazziDebug`	`./gradlew testDebugUnitTest` after adding `roborazzi.test.compare=true` to your gradle.properties file. or `./gradlew testDebugUnitTest -Proborazzi.test.compare=true`	Review changes made to an image. This action will compare the current image with the saved one, generating a comparison image labeled as `[original]_compare.png`. It also produces a JSON file containing the diff information, which can be found under `build/test-results/roborazzi`.
`./gradlew verifyRoborazziDebug`	`./gradlew testDebugUnitTest` after adding `roborazzi.test.verify=true` to your gradle.properties file. or `./gradlew testDebugUnitTest -Proborazzi.test.verify=true`	Validate changes made to an image. If there is any difference between the current image and the saved one, the test will fail.
`./gradlew verifyAndRecordRoborazziDebug`	`./gradlew testDebugUnitTest` after adding `roborazzi.test.verify=true` and `roborazzi.test.record=true` to your gradle.properties file. or `./gradlew testDebugUnitTest -Proborazzi.test.verify=true -Proborazzi.test.record=true`	This task will first verify the images and, if differences are detected, it will record a new baseline.
`./gradlew clearRoborazziDebug`	This is not a test task.	Note: This is an experimental task. This task will clear the saved images. This task also deletes the cached images. Please be careful when using this task.

The comparison image, saved as [original]_compare.png, is shown below:

You can check the test report in build/reports/roborazzi/index.html

This uses JetNew from Compose Samples. You can check the pull request introducing Roborazzi to the compose-samples here.

Add dependencies

Description	Dependencies
Core functions	`testImplementation("io.github.takahirom.roborazzi:roborazzi:[version]")`
Jetpack Compose	`testImplementation("io.github.takahirom.roborazzi:roborazzi-compose:[version]")`
JUnit rules	`testImplementation("io.github.takahirom.roborazzi:roborazzi-junit-rule:[version]")`

How to use

Take a screenshot manually

You can take a screenshot by calling captureRoboImage().

app/src/test/java/../ManualTest.kt

import androidx.compose.ui.test.junit4.createAndroidComposeRule
import androidx.test.core.app.ActivityScenario
import androidx.test.espresso.Espresso.onView
import androidx.test.ext.junit.runners.AndroidJUnit4
import org.junit.Test
import org.junit.runner.RunWith
import org.robolectric.annotation.GraphicsMode

// All you need to do is use the captureRoboImage function in the test!
import com.github.takahirom.roborazzi.captureRoboImage


// Tips: You can use Robolectric while using AndroidJUnit4
@RunWith(AndroidJUnit4::class)
// Enable Robolectric Native Graphics (RNG) 
@GraphicsMode(GraphicsMode.Mode.NATIVE)
class ManualTest {
  @get:Rule
  val composeTestRule = createAndroidComposeRule<MainActivity>()

  @Test
  fun captureRoboImageSample() {
    // Tips: You can use Robolectric with Espresso API
    // launch
    ActivityScenario.launch(MainActivity::class.java)

    // Capture screen
    onView(ViewMatchers.isRoot())
      // If you don't specify a screenshot file name, Roborazzi will automatically use the method name as the file name for you.
      // The format of the file name will be as follows:
      // build/outputs/roborazzi/com_..._ManualTest_captureRoboImageSample.png
      .captureRoboImage()

    // Capture Jetpack Compose Node
    composeTestRule.onNodeWithTag("AddBoxButton")
      .onParent()
      .captureRoboImage("build/compose.png")
  }
}

Roborazzi supports the following APIs.

Capture	Code
✅ Jetpack Compose's onNode()	`composeTestRule.onNodeWithTag("AddBoxButton") .captureRoboImage()`
✅ Espresso's onView()	`onView(ViewMatchers.isRoot()) .captureRoboImage()` `onView(withId(R.id.button_first)) .captureRoboImage()`
✅ View	`val view: View = composeTestRule.activity.findViewById<View>(R.id.button_second) view.captureRoboImage()`
✅ Jetpack Compose lambda	`captureRoboImage() { Text("Hello Compose!") }`
Experimental🧪 ✅ Captures the entire screen, including dialogs	`captureScreenRoboImage()`
✅ Bitmap	`val bitmap: Bitmap = createBitmap(100, 100, Bitmap.Config.ARGB_8888) .apply { applyCanvas { drawColor(android.graphics.Color.YELLOW) } } bitmap.captureRoboImage()`

Device configuration

You can configure the device by using the @Config annotation and RobolectricDeviceQualifiers.

Configuration	Code
✅ Predefined device configuration	You can change the device configuration by adding `@Config` to the class or method. `@RunWith(AndroidJUnit4::class) @GraphicsMode(GraphicsMode.Mode.NATIVE) @Config(qualifiers = RobolectricDeviceQualifiers.Pixel5) class RoborazziTest {` `@Test @Config(qualifiers = RobolectricDeviceQualifiers.Pixel5) fun test() {`
✅ Night mode	`@Config(qualifiers = "+night")`
✅ Locale	`@Config(qualifiers = "+ja")`
✅ Screen size	`@Config(qualifiers = RobolectricDeviceQualifiers.MediumTablet)`

Integrate to your GitHub Actions

It is easy to integrate Roborazzi to your GitHub Actions.

Add a job to store screenshots

name: store screenshots

on:
  push

env:
  GRADLE_OPTS: "-Dorg.gradle.jvmargs=-Xmx6g -Dorg.gradle.daemon=false -Dkotlin.incremental=false"

jobs:
  test:
    runs-on: macos-latest

    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-java@v3.9.0
        with:
          distribution: 'zulu'
          java-version: 19

      - name: Gradle cache
        uses: gradle/gradle-build-action@v2

      - name: test
        run: |
          # Create screenshots
          ./gradlew app:recordRoborazziDebug --stacktrace

      # Upload screenshots to GitHub Actions Artifacts
      - uses: actions/upload-artifact@v3
        with:
          name: screenshots
          path: app/build/outputs/roborazzi
          retention-days: 30

Add a job to verify screenshots

name: verify test

on:
  push

env:
  GRADLE_OPTS: "-Dorg.gradle.jvmargs=-Xmx6g -Dorg.gradle.daemon=false -Dkotlin.incremental=false"

jobs:
  test:
    runs-on: macos-latest

    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-java@v3.9.0
        with:
          distribution: 'zulu'
          java-version: 19

      - name: Gradle cache
        uses: gradle/gradle-build-action@v2

      # Download screenshots from main branch
      - uses: dawidd6/action-download-artifact@v2
        with:
          name: screenshots
          path: app/build/outputs/roborazzi
          workflow: test.yaml
          branch: main

      - name: verify test
        id: verify-test
        run: |
          # If there is a difference between the screenshots, the test will fail.
          ./gradlew app:verifyRoborazziDebug --stacktrace

      - uses: actions/upload-artifact@v3
        if: ${{ always() }}
        with:
          name: screenshot-diff
          path: app/build/outputs/roborazzi
          retention-days: 30

      - uses: actions/upload-artifact@v3
        if: ${{ always() }}
        with:
          name: screenshot-diff-reports
          path: app/build/reports
          retention-days: 30

      - uses: actions/upload-artifact@v3
        if: ${{ always() }}
        with:
          name: screenshot-diff-test-results
          path: app/build/test-results
          retention-days: 30

Advanced workflow Sample: Compare Snapshot Results on Pull Requests

For those who are looking for a more advanced example, we have prepared a sample repository that demonstrates how to use Roborazzi to compare snapshot results on GitHub pull requests. This sample showcases the integration of Roborazzi with GitHub Actions workflows, making it easy to visualize and review the differences between snapshots directly in the pull request comments.

Check out the roborazzi-compare-on-github-comment-sample repository to see this powerful feature in action and learn how to implement it in your own projects.

Example of the comment

RoborazziRule (Optional)

RoborazziRule is a JUnit rule for Roborazzi. RoborazziRule is optional. You can use captureRoboImage() without this rule.

RoborazziRule has two features.

Provide context such as RoborazziOptions and outputDirectoryPath etc for captureRoboImage().
Capture screenshots for each test when specifying RoborazziRule.options.captureType.

For example, The following