Simon Willison Evaluates Webframe As Scraper And Sandbox
Webframe, a browser-JavaScript runner, shows promise for scraping but has rough edges. Its performance is comparable to shot-scraper, according to Simon Willison.
Original reporting · simonwillison.net
Original: “webframe: evaluated as a scraper and a sandbox”
Read “webframe: evaluated as a scraper and a sandbox” at Simon Willison →How this was written. Content Memory wrote this article with a language model from Simon Willison’s reporting (model: @cf/meta/llama-3.3-70b-instruct-fp8-fast); it describes that reporting rather than adding to it. The original, linked above, is the full story.
Key points from Simon Willison’s report
- Webframe is a fast and lightweight browser-JavaScript runner suitable for scraping
- Webframe has rough edges, including CSP failures and limited browser controls
- Webframe's performance is comparable to shot-scraper, another JavaScript tool
Simon Willison reported on the evaluation of webframe, a project that allows users to run a web page like a program, according to its creator Jeff Lindsay. Webframe executes a web page in a hidden headless web view and outputs console.log() to standard output and exceptions to standard error. This functionality makes it suitable for scraping, a key use case for the tool.
Webframe is written in Go and built on top of the glaze library, which provides access to a system browser, eliminating the need to bundle its own browser engine. The glaze library supports various browsers, including WKWebView on macOS, WebKitGTK on Linux, and WebView2 on Windows. This approach keeps the package small and lightweight.
The evaluation found webframe to be fast and lightweight, with performance comparable to shot-scraper, another JavaScript tool. However, webframe has some important rough edges, including CSP failures, a 0×0 viewport, hanging loads, missing HTTP metadata, and limited browser controls. These issues may impact its usability and effectiveness for certain tasks.
Despite its limitations, webframe shows promise as a scraping tool, particularly given its small package size and lack of dependence on Playwright. Its ability to run a web page like a program makes it an interesting option for developers looking for a lightweight solution. Further development and refinement may help address the current rough edges and improve its overall performance.
The comparison to shot-scraper is notable, as it highlights webframe's potential as a viable alternative. Shot-scraper is a JavaScript tool developed by Simon Willison, and webframe's ability to match its performance is a significant achievement. However, the differences in dependencies and package size may make webframe a more attractive option for certain use cases.
Looking ahead, it will be interesting to see how webframe develops and whether it can address its current limitations. With further refinement, webframe may become a more robust and reliable option for scraping and other tasks. Its unique approach and small package size make it a project worth watching in the future.
The evaluation of webframe serves as a reminder of the importance of ongoing development and refinement in the field of web scraping and browser automation. As new tools and projects emerge, they must be carefully evaluated and compared to existing solutions to determine their strengths and weaknesses. This process helps drive innovation and improvement in the field.
Questions and answers
What is webframe and what does it do?
Webframe is a project that lets you run a web page like a program, executing it in a hidden headless web view and outputting console.log() to standard output and exceptions to standard error.
How does webframe compare to shot-scraper?
Webframe's performance is comparable to shot-scraper, but it has a smaller package size and does not depend on Playwright.
What are the rough edges of webframe?
Webframe has several rough edges, including CSP failures, a 0×0 viewport, hanging loads, missing HTTP metadata, and limited browser controls.
