autotab-starter VS Playwright

You can pause recording at anytime. For passwords we recommend using the login tools in our starter repo, check out the example config here: https://github.com/Planetary-Computers/autotab-starter/blob/...

For sensitive information that appears in the DOM during recording, there is a chance it could be included in a prompt to the LLM. We're using OpenaAI via API, which is SOC 2 & 3 compliant and does not use data for model training (supposedly) https://trust.openai.com

/*Importing test which help you to write test cases */ /*Importing expect which help to validate expected result */ const { test, expect } = require('@playwright/test'); /* Open Playwright url*/ test('has title', async ({ page }) => { await page.goto('https://playwright.dev/'); // Expect a title "to contain" a substring. await expect(page).toHaveTitle(/Playwright/); }); test('get started link', async ({ page }) => { await page.goto('https://playwright.dev/'); // Click the get started link. await page.getByRole('link', { name: 'Get started' }).click(); // Expects page to have a heading with the name of Installation. await expect(page.getByRole('heading', { name: 'Installation' })).toBeVisible(); });

In this tutorial, we can extract data from the HTML structure, so we will go with Cheerio, but for extracting data from SPAs or JavaScript-rendered websites, Crawlee also supports headless browser libraries like Playwright and Puppeteer

Playwright is an end-to-end testing framework developed by Microsoft and available in multiple programming languages. Its focus is on cross-browser testing, using Chromium as the default browser. To perform the test logic on a Chromium-based browser, it controls and instructs a browser instance to perform desired actions via the DevTools Protocol.

Playwright documentation

In Crawlee, you can scrape JavaScript rendered websites using the built-in headless Puppeteer and Playwright browsers. It is important to note that, by default, Crawlee scrapes in headless mode. If you don't want headless, then just set headless: false.

Playwright, an end-to-end test runner;

We start with a project that was bootstrapped with npx create-next-app. For the E2E test we use Playwright and set it up as described in the testing guide provided by Next.js.

Playwright is a powerful tool developed by Microsoft, it allows developers to write reliable end-to-end tests and perform browser automation tasks with ease. What sets Playwright apart is its ability to work seamlessly across multiple browsers (Chrome, Firefox, and WebKit), it provides a consistent and efficient way to interact with web pages, extract data, and automate repetitive tasks. Moreover, it supports various programming languages such as Node.js, Python, Java, and .NET, that’s making it a versatile choice for web scraping projects. Whether you're scraping public data for analysis, building a web crawler, or automating manual workflows, Playwright has you covered.

The consensus I could gather is either use playwright or use a workaround to solve it in the puppeteer layer. The root cause of the bug is a websocket size limitation on the CDP protocol for chromium.