Jest Is No Longer My Go-To Testing Framework

In the tech world, Jest is the go-to testing framework for many projects. It’s easy to set up, integrates nicely with your codebase, and can get the job done pretty well. That’s also the reason why all the project starter templates I have created have Jest baked into it, with other auxiliary libraries such as jest-mock-extended to make mocking services for unit tests (UTs) a breeze. But, as our test-suites started to grow in different projects, people started to complain about how long it takes for tests to run. The issue is exacerbated by the fact that the projects also have husky installed, and as part of the pre-commit routine, it runs at least the UTs and some integration tests. In essence, Jest was getting the job done, but it was behaving like a reluctant teenager asked to do chores.

I initially ignored the problem for two reasons:

1. I have more important tasks on my plate to go through on a daily basis, and refactoring our entire test suite to optimize its duration wasn’t the best use of my time
2. I am lazier than a sloth on a holiday

While you could ask “Shouldn’t those reasons be flipped?”, my answer to that would be “Erm, let’s not get hung up on details”. However, the issue kept living rent-free in my head long enough that I rolled up my sleeves on one rainy weekend where Netflix had nothing new to offer, played “Blinding Lights by The Weeknd” on Spotify, and started looking into ways to speed up a test suite in one of my projects.

Project Test Statistics

The project in question is a fairly new Nest.js + Mikro ORM project, however it has a decent amount of modules. We don’t have a lot of UTs because we only write UTs when it provides us value (e.g. helper methods), but the E2E test suite is fairly comprehensive. Here’s how long it takes on my machine for the test suites to run with Jest:

My PC Configuration

E2E and Integration Test Suite

UTs

With my baseline in-place, I started looking for alternatives and landed on Vitest.

Why I Chose Vitest

If I was writing a Technical Decision Record (TDR), in this section I would have to justify my choice by comparing with other testing frameworks. However, I am doing this on a weekend, so I wouldn’t really go that route. But, I can offer the following reasons:

1. I wanted something that would require least effort to migrate to, and Vitest offers very similar APIs to Jest. They even offer a forked version of jest-mock-extended, dubbed vitest-mock-extended with basically identical APIs.
2. Nest.js already has a decent documentation on how to integrate with Vitest
3. I have already seen countless comparison posts (e.g. this one by James Milner) between testing frameworks, and Vitest has always stood out as the winner in those benchmarks.
4. I have tried @swc/jest in the past, but that exhibited some funky behavior with decorators and metadata (in particular, with Mikro ORM), and I haven’t found a fix for it. However, at this point, I wasn’t sure that Vitest would work either.

Getting My Feet Wet by Refactoring Unit Tests

Since I wasn’t even sure if Vitest would work, I decided to start off by refactoring our UTs. I installed Vitest and the other necessary libraries:

yarn add -D vitest unplugin-swc @swc/core @vitest/coverage-v8 vitest-mock-extended

Then, I created the config for Vite (vite.config.mts)

import swc from 'unplugin-swc';
import { defineConfig } from 'vitest/config';

export default defineConfig({
  test: {
    globals: true,
    root: './',
    include: ['**/*.spec.ts', '!**/*.int.spec.ts']
  },
  resolve: {
    alias: {
      '@/test/': new URL('./test/', import.meta.url).pathname,
      '@/': new URL('./src/', import.meta.url).pathname
    }
  },
  plugins: [swc.vite()]
});

And a config for SWC (.swcrc)

{
  "jsc": {
    "parser": {
      "syntax": "typescript",
      "tsx": false,
      "decorators": true
    },
    "target": "es2020",
    "transform": {
      "legacyDecorator": true,
      "decoratorMetadata": true
    }
  },
  "module": {
    "type": "es6"
  },
  "minify": false
}

Then, I had to do two things:

1. Replace jest-mock-extended imports with vitest-mock-extended
2. Replace any references of jest with vi, e.g. jest.fn() would become vi.fn()

There were a few cases where the mapping wasn’t exactly one-to-one, but not a big deal either.

// Jest
jest.useFakeTimers({ doNotFake: ['nextTick'] }).setSystemTime(new Date('2022-01-01'));

// Vitest
vi.useFakeTimers({ shouldAdvanceTime: true }).setSystemTime(new Date('2022-01-01'));

// Jest
(fs.readFileSync as jest.Mock).mockReturnValue(...);

// Vitest
// import { vi, Mock } from "vitest"
(fs.readFileSync as Mock).mockReturnValue(...);

Finally, I added a script in the package.json:

"test:vitest": "dotenv -e ./.env.test.local -- vitest run",

With all of that done, I ran the unit test suite and couldn’t believe the results. It took 1.51s, almost a 10x improvement over 11.417s with Jest!

Excited, I moved on to refactor the integration and E2E tests with Vitest.

Refactoring Integration and E2E Tests with Vitest

The process is somewhat similar to the previous section. I created a new config file for integration and E2E tests:

vitest.e2e.config.mts

import swc from 'unplugin-swc';
import { defineConfig } from 'vitest/config';

export default defineConfig({
  test: {
    globals: true,
    root: './',
    include: ['**/*.int.spec.ts', '**/*.e2e-spec.ts'],
    isolate: false,
    fileParallelism: false,
    disableConsoleIntercept: true
  },
  resolve: {
    alias: {
      '@/test/': new URL('./test/', import.meta.url).pathname,
      '@/': new URL('./src/', import.meta.url).pathname
    }
  },
  plugins: [swc.vite()]
});

A few notes about the config above:

1. Vitest docs suggest that we disable isolate for faster tests
2. In our E2E tests, we truncate the database at the end of the test suite, which would cause interference with other test suites. So I turned off fileParallelism, which is equivalent to running Jest tests with --max-workers=1.
3. Vitest swallows console.log outputs by default, hence I enabled disableConsoleIntercept.

After that, it was the same process as described above, where I had to replace instances of jest with vi. I ran the E2E test suite and I expected the thing to completely blow up in my face. And that’s exactly what happened, all tests failed. And thus I embarked on a journey to overcome a few roadblocks iteratively.

Roadblock 1: TypeError: Unknown file extension ".ts" for /path/to/file.entity.ts

All the tests failed with this exact error, where the transpiler seemingly couldn’t understand .ts extension for Mikro ORM entities. This is a known issue and I wouldn’t have faced it had I read the documentation beforehand, but that would’ve been very out of character as a developer of me. However, the fix was a single line change in the Mikro ORM config I have:

...(process.env.NODE_ENV === "test" ? { dynamicImportProvider: (id) => import(id) } : {}),

With that fix, some of the tests started running and passing! However, I ran into some other roadblocks.

Roadblock 2: BASE_URL environment variable

I noticed some tests that involved constructing a URL object and calling mock endpoints of third-party APIs were failing. After a bit of good ol’ fashioned console.log-s, I discovered the issue: even though I explicitly set an environment variable BASE_URL in my .env.test.local file, it was being overridden to a different value by Vitest. Luckily, there is already a GitHub issue that outlines a solution for it, so all I had to do was update my Vite config files so that Vite would not override the variable.

test: {
  ...
  env: {
    BASE_URL: process.env["BASE_URL"] ?? "/",
  },
},

Roadblock 3: Optional @Query parameters in controllers

This one took the longest for me to figure out. In one of the controllers, there is an optional query param:

@UseGuards(JwtAuthGuard, RolesGuard)
@Roles(EUserRole.USER)
@Get()
async findAll(
  @CurrentUser() currentUser: ITokenizedUser,
  @Query("status", new ParseEnumPipe(EStatus, { optional: true })) status?: EStatus,
): Promise<FindAllResponse[]> {
  ...
}

And some of the tests associated to this endpoint were failing with “Internal Server Error”. The failing tests had one thing common among themselves: they did not pass any value for the optional query param status. The error was a very obfuscated error, that led me to think that transpilation was going wrong at some point:

[Nest] 69363  - 09/15/2024, 5:10:47 PM   ERROR [ExceptionsHandler] [4ecce208930a9ea8f52c0251ee530910] Cannot read properties of undefined (reading 'constructor')
TypeError: Cannot read properties of undefined (reading 'constructor')
    at MetadataStorage.getAncestors (/home/mahieyinrahmun/.../project/node_modules/src/MetadataStorage.ts:244:64)
    at MetadataStorage.getMetadata (/home/mahieyinrahmun/.../project/node_modules/src/MetadataStorage.ts:178:33)
    at MetadataStorage.getExposedMetadatas (/home/mahieyinrahmun/.../project/node_modules/src/MetadataStorage.ts:108:17)
    at MetadataStorage.getExposedProperties (/home/mahieyinrahmun/.../project/node_modules/src/MetadataStorage.ts:116:17)
    at TransformOperationExecutor.getKeys (/home/mahieyinrahmun/.../project/node_modules/src/TransformOperationExecutor.ts:461:54)
    at TransformOperationExecutor.transform (/home/mahieyinrahmun/.../project/node_modules/src/TransformOperationExecutor.ts:150:25)
    at ClassTransformer.plainToInstance (/home/mahieyinrahmun/.../project/node_modules/src/ClassTransformer.ts:77:21)
    at Object.plainToClass (/home/mahieyinrahmun/.../project/node_modules/src/index.ts:71:27)
    at ValidationPipe.transform (/home/mahieyinrahmun/.../project/node_modules/@nestjs/common/pipes/validation.pipe.js:60:39)
    at /home/mahieyinrahmun/.../project/node_modules/@nestjs/core/pipes/pipes-consumer.js:16:33

Another thing I noticed is that I have used the exact same syntax for the query param in other controllers as well, the key difference being that they were not optional params. With all those clues, I decided to convert the query param to a DTO:

export class FindAllQueryParams {
  @IsOptional()
  @IsEnum(EStatus)
  status?: EStatus;
}

@UseGuards(JwtAuthGuard, RolesGuard)
@Roles(EUserRole.USER)
@Get()
async findAll(
  @CurrentUser() currentUser: ITokenizedUser,
  @Query() { status }: FindAllQueryParams,
): Promise<FindAllResponse[]> {
  ...
}

And voila, the tests started to pass!

Benchmarks

Next, I decided to do some benchmarks. I ran the test suites 5 times with Jest and Vitest, noting down the time it took for the tests to complete. Then, I took the average of the runs. Note that I cleared out caches of Jest and Vitest before each test run. Here are the findings:

As you can see, Vitest consistently is faster than Jest by large margins in UTs, and saves about 24 seconds for E2E and Integration Tests.

Conclusion

At the expense of half a day of my weekend, now our test suite is way faster. The numbers speak for themselves, and I can bask in the glory of a worthwhile refactor that went right. In all seriousness- well, semi-seriousness-this migration has been a game-changer: our tests are faster, meaning our developers can commit code faster than before. Another consideration is that machines in CI environments are not as powerful as my work laptop, so faster tests would save quite some build minutes at e.g. GitHub Actions. So next time you’re procrastinating on some task because it’s not “the best use of your time” or you’re just embracing your inner sloth, you may want to recall the accomplishment you read about above. Who knows? Maybe your next “I’ll do it later” project might just turn into the “How did we live without this?” solution.