6 Open Source Tools That Give You the Web Back

Hello, I'm Maneshwar. I'm building git-lrc, a Micro AI code reviewer that runs on every commit. It is free and source-available on Github. Star git-lrc to help devs discover the project. Do give it a try and share your feedback.

You've probably noticed this if you've tried to fetch a public webpage in the last two years.

You write four lines of Python. You get a 403. You add a User-Agent header. You get a 403 with better manners.

You spin up headless Chrome and get a challenge page that spins forever.

Eventually you go looking at pricing pages for data APIs, where a cheerful landing page offers to sell you back the exact same public HTML for $100 a month.

Meanwhile the big labs vacuumed up that same web at a scale none of us will ever match, and now the standard way to read a page programmatically is to pay somebody.

The web went from "open by default" to "open, but only if you look like a person holding a mouse."

So let's talk about eight open source projects quietly clawing that access back. I've used most of these in anger. Some are brilliant.

Quick note on star counts: I mention a few, because they are a decent signal of "somebody else has hit the bugs before you." They are not a signal of maintenance. A repo can have 70k stars and a last commit from 2023. Always check the commit graph before you check the star count.

1. Firecrawl: the one that turned a URL into a prompt

What it does: you hand it a URL, it hands you clean markdown. Not HTML with the nav bar and the cookie banner and four newsletter modals. Markdown. Headings, paragraphs, links, done.

It also does more than single pages.

The crawl endpoint walks a whole site, map gives you a URL inventory without fetching everything, and structured extraction pulls typed JSON out of a page if you describe the shape you want.

from firecrawl import Firecrawl
app = Firecrawl(api_key="fc-...")
doc = app.scrape("https://example.com/docs/getting-started",
                 formats=["markdown"])
print(doc.markdown)

Reach for it when: you want the web as text, you want it now, and you would rather ship the feature than build a crawling team.

2. Crawl4AI: the same idea, but yours

What it does: an open source, async Python crawler that outputs LLM-ready markdown.

No API key. No credits. No rate limit other than the one your target site imposes and the one your conscience should.

import asyncio
from crawl4ai import AsyncWebCrawler
async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url="https://example.com")
        print(result.markdown)
asyncio.run(main())

Reach for it when: you have volume, you have infra people, or you have data you cannot legally hand to a third party service.

Also, both of these projects have converged on the same insight, which is fun to watch.

3. Browser-use: an agent with hands

What it does: wires an LLM to a real browser and lets it drive. Clicking, typing, scrolling, filling forms, logging in, navigating a multi step checkout.

You describe the goal in English, it figures out the DOM.

Reach for it when: the target is genuinely interactive, the volume is low, and the alternative is a human doing it by hand.

---

What can Browser Use do?

Browser Use lets an AI agent use a web browser the same way you do — it opens pages, clicks buttons, types, and fills in forms. You describe the task, and it completes it. For example, you can have it:

📋 Fill Forms

Task: "Fill in this job application with my resume and information."

Example code ↗

🍎 Extract data

Task: "Extract structured data about my followers and export it as a CSV."

extract-followers.mp4

Browser Use Cloud Docs ↗

💻 QA Automation

Task: "QA test my local website and report any bugs, usability issues, and visual inconsistencies."

Browser Use CLI ↗

Quickstart

If you want to use Browser Use in your agent (Claude Code, Codex, Cursor, Hermes, OpenClaw, etc.), paste this prompt, and it sets everything up itself:

Install or upgrade browser-use to the latest stable version with uv using Python 3.12, run `browser-use skill

…

4. Scrapy: the one that was doing this before it was cool

What it does: the Python crawling framework. Request scheduling, concurrency, retries, middleware, item pipelines, autothrottle, the works.

It has been in production since roughly 2008 and it has crawled more pages than every other tool on this list combined.

Reach for it when: high volume, stable, mostly static targets. It is the boring correct answer and boring correct answers pay the bills.

pip install scrapy

5. Crawlee: the infrastructure you were about to rebuild badly

What it does: the Node.js (and now Python) crawling library from the Apify folks.

Proxy rotation, session management, browser fingerprint spoofing, automatic retries with backoff, request queues that survive a restart, and a unified API whether you are using plain HTTP or Playwright or Puppeteer under the hood.

Reach for it when: you are in the Node ecosystem, or your targets are actively hostile and you need the anti-blocking stack without paying for one.

6. Scrapling: the scraper that heals itself

What it does: a Python scraper that remembers what an element was, so when the site redesigns and your .product-title becomes .ProductCard__title--x7f2, it re-locates the element by similarity instead of returning an empty list.

Reach for it when: you maintain many spiders over a long time and selector rot is your actual bottleneck.

from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, DynamicFetcher
StealthyFetcher.adaptive =

About that "sell this as a service for $5,000 a month" thing

Every video on this topic ends with the same pitch: lead gen agencies charge thousands a month for this data, your cost is zero, go get rich.

I want to be careful here, because there is a real business in web data, but "your cost is zero" is doing an enormous amount of work in that sentence.

Your cost is not zero.

Your cost is residential proxies, which are the single biggest line item and which no open source project ships.

Your cost is CAPTCHA solving.

Your cost is the engineer-hours when four targets redesign in the same week.

Your cost is storage, monitoring, and the pager.

What those agencies are actually charging for is not the scraping code, which as you have just seen is free and excellent.

They are charging for the operational layer and the guarantee that the data shows up on Monday even though the site changed on Sunday.

That is a real service and it is worth real money.

Just go in knowing what you are actually selling.

The boring section that keeps you out of trouble

You knew this was coming.

Being able to scrape something is not the same as being allowed to, and the gap between those two is where people get their infrastructure banned or their company sued.

A short, non-lawyer summary of what I actually do:

• Read robots.txt and honour it. Beyond ethics, it is self-interested: polite crawlers survive longer. Aggressive ones get IP ranges blocked, and then nobody on your team can scrape that domain.
• Rate limit yourself before they rate limit you. Concurrency of 2 and a delay is almost always enough. Nobody needs 200 requests a second against a small site's origin server.
• Check the Terms of Service, especially for anything behind a login. "Publicly accessible" and "you have permission" are different claims.
• Personal data is a different legal universe. GDPR, CCPA and friends do not care that the data was on a public page. Scraping product prices and scraping names with email addresses are not the same activity.

Seriously. Open devtools, filter the network tab by Fetch/XHR, and look.

Roughly a third of the scrapers I have written could have been three lines against a JSON endpoint that was sitting there the whole time.

Scraping is what you do after that fails, not instead of checking.

Wrapping up

Six projects, one idea: the web is still readable, the tooling to read it is free and open, and most of the wall that showed up over the last two years is made of TLS fingerprints and inertia rather than actual locks.

Pick by job, not by star count. Start light. Check for an API. Be a polite guest on other people's servers.

And if you only take one thing from this: next time you get an inexplicable 403, before you reach for the browser, try impersonate="chrome". You will feel very silly and very happy.

What did I miss? I know some of you have a favourite obscure Go crawler you have been waiting for an excuse to talk about. Drop it in the comments, I read all of them.

AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs — without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

⭐ Star it on GitHub: