Playwright makes end-to-end browser testing fun

Playwright is a library, developed by Microsoft, for writing end-to-end tests for interactive web applications. It serves a similar purpose as Puppeteer yet I found it much more enjoyable to use, especially in ways in which I struggled with Puppeteer. It is more extensive in-scope, for example it has a tightly integrated test-runner, while also usable through external ones like Jest.

First let's look at the problem being solved by both Ps. Most modern UI toolkits run on asynchronous programming which unlocks one of the greatest general improvements in UX, namely UIs not freezing and being more available for user interaction. The downside is that you now have to think about race conditions in your UI code. React has popularized patterns for taming the async beast on the development side but testing it is a different pair of shoes.

For server rendered pages end-to-end tests could be as simple as checking for the occurrence of a string inside the generated HTML. For interactive parts of the page, like forms, you would, in my experience, re-hand-craft those requests and once again check for a string inclusion.

We can not do that for asynchronous interactive client-heavy pages. There are no immediately searchable strings, but rather components "pop into view" whenever they are ready. The way both Puppeteer and Playwright solve it is by offering timeout based APIs which wait until whatever you are looking for enters the page.

expect(
  await page.waitForSelector("text=Some text")
).toBeTruthy();

Now there are all kinds of things that can go wrong here and debugging that is where Playwright shines.

My favorite debugging features

Inspector

Playwright comes with its own inspector UI which you can use with whatever browsers engine you like (oh I forgot to mention, Playwright supports all modern browser engines). Like a debugger the inspector allows you to step through your tests line by line, while another window with your chosen browser opens up, reflecting the current state of the tested page. You can also open up that browser's dev tools to really go into the details of what is going on.
This is what it looks like:

Now what can kind of voodoo hackery do you have to perform to get there? Start your tests with PWDEBUG=1. That's it.

Error messages

If you are like me, you probably don't want to start a debugger UI for every issue and that's fine. The Playwright developers have got our collective backs and also added super helpful error messages. Let's look at what happens to our first example when the text we are selecting for is not visible.

Playwright does not only throw an error informing us that the test timed out, but also tells us what it was fruitlessly waiting for:

waiting for selector "text=Some text"
  selector resolved to hidden <span>Some text</span>
waiting for element to be visible, enabled and stable
    element is not visible - waiting...

Screenshots

Visual comparison tests (often used as regression tests) are a great help in frontend development. This can also be accomplished with other tools like Puppeteer but I do consider this feature essential enough for end-to-end testing that I appreciate Playwright supporting it out-of-the-box. To get a snapshot-based visual regression test going all you'd have to write is (example straight from the playwright docs):

test("example test", async ({ page }) => {
  await page.goto("https://playwright.dev");
  expect(
    await page.screenshot()
  ).toMatchSnapshot("landing.png");
});

Okay I lied a little there. You are most likely using MacOS and your CI is running Linux and even though you can use the same browser engine, rendering will be ever so slightly pixel-different enough that snapshots created in your OS won't pass in CI.

So for updating snapshots from your dev machine you will want to have Docker at hand. I have asked about alternative ways of doing that in Playwright's Slack, and it sounds like they might be working on something there. I have no idea what it will be, but given the current DX niceties I have good faith they might come up with a better solution than Docker here.

Ehh wasn't I talking about debugging?

Right, and for that we can quickly forget about Docker again (anxiety--). You can configure Playwright to take screenshots whenever tests fails. Combine that with GitHub Actions' ability to upload artifacts and you can quickly find out why tests are failing in CI. For GitHub Actions it would look something like this:

- uses: actions/upload-artifact@v2
  if: failure()
  with:
    name: Test Results
    path: test-results/
    if-no-files-found: ignore

GitLab also support artifacts but I have not yet found out how to have it not error when a test succeeds (and the test-results/ dir does not exist).

Alternatives

There is a lineage of end-to-end testing libraries I have neglected to mention here. Selenium popularized the idea of just using a real browser in your testing environment, through testing-library I first encountered APIs very similar to what we use with both Puppeteer and Playwright... and there are many more before and between all of those, but I do have to say that Playwright gets DX right in a way I have not encountered before.