[BUG] throwing from pageerror callback leads to unpredictable behavior

[bahmutov]

System info

• Playwright Version: [v1.39]
• Operating System: [Mac 14.0]
• Browser: [Chromium]
• Other info: UI mode

Source code

• I provided exact source code that allows reproducing the issue locally.

I am trying to fail the test if the page throws any errors.

page.on('pageerror', (exception) => {
  console.log('page error!')
  throw exception
})
await page.goto('/')
await expect(page.locator('.todo-list li')).toHaveCount(0)

The UI test mode shows really unpredictable behavior, see the video below. Every time I click on the run button, it does something differently, and sometimes the results disappear.

throw-on-pageerror.mp4

Even when it does fail due to the li element count, it does not retry for 5 seconds, but only for 50-100ms because that's when the exception arrives.

If I try to use expect to fail the test if fails the assertion, but proceeds

page.on('pageerror', async (exception) => {
  console.log('page error!')
  expect(false, 'page error').toBeTruthy()
})
await page.goto('/')
await expect(page.locator('.todo-list li')).toHaveCount(0)

Expected

The test fails if the pageerror callback throws an error

Actual

[Describe actual behavior]

[marko-simic]

Perhaps you can consider this approach.

test("Example", async ({ page }) => {
  // Log all uncaught errors to the terminal
  page.on("pageerror", (exception) => {
    expect(exception, exception).not.toBeDefined();
  });

  // Navigate to a page with an exception.
  await page.goto('data:text/html,<script>throw new Error("Test")</script>');
});

[dgozman]

I can partly reproduce the issue with the following snippet. When running in UI mode, I don't see the "Some error", but instead see the expect's error.

import { test, expect } from '@playwright/test';

test('a test', async ({ page }) => {
  await page.goto('https://playwright.dev');
  setTimeout(() => { throw new Error('Some error') }, 500);
  await expect(page.locator('.todo-list li')).toHaveCount(10);
});

However, I cannot reproduce the "empty result" issue. @bahmutov Could you please share the full repro, with the actual page url, that triggers the empty result? I suspect it is timing-related.

[bahmutov]

Sure, here is a standalone repo https://github.com/bahmutov/pw-on-error
It acts very weirdly, showing after hooks run before actions (and clicking on the actions shows nothing) I suspect the thrown exception has the test finish immediately, but the actions running in parallel seem to be not cleaned up

[dgozman]

@bahmutov Thank you for the repro. I can observe the same: expect actions at the end with an empty log. However, both expects reliably succeed in my case, I guess due to different timing.