On this tutorial, we exhibit the best way to harness Crawl4AI, a contemporary, Python‑primarily based net crawling toolkit, to extract structured knowledge from net pages straight inside Google Colab. Leveraging the ability of asyncio for asynchronous I/O, httpx for HTTP requests, and Crawl4AI’s constructed‑in AsyncHTTPCrawlerStrategy, we bypass the overhead of headless browsers whereas nonetheless parsing advanced HTML by way of JsonCssExtractionStrategy. With just some strains of code, you put in dependencies (crawl4ai, httpx), configure HTTPCrawlerConfig to request solely gzip/deflate (avoiding Brotli points), outline your CSS‑to‑JSON schema, and orchestrate the crawl by means of AsyncWebCrawler and CrawlerRunConfig. Lastly, the extracted JSON knowledge is loaded into pandas for instant evaluation or export.
What units Crawl4AI aside is its unified API, which seamlessly switches between browser-based (Playwright) and HTTP-only methods, its strong error-handling hooks, and its declarative extraction schemas. In contrast to conventional headless-browser workflows, Crawl4AI means that you can select essentially the most light-weight and performant backend, making it preferrred for scalable knowledge pipelines, on-the-fly ETL in notebooks, or feeding LLMs and analytics instruments with clear JSON/CSV outputs.
!pip set up -U crawl4ai httpxFirst, we set up (or improve) Crawl4AI, the core asynchronous crawling framework, alongside HTTPX. This high-performance HTTP consumer offers all of the constructing blocks we want for light-weight, asynchronous net scraping straight in Colab.
import asyncio, json, pandas as pd
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, HTTPCrawlerConfig
from crawl4ai.async_crawler_strategy import AsyncHTTPCrawlerStrategy
from crawl4ai.extraction_strategy import JsonCssExtractionStrategyWe usher in Python’s core async and knowledge‑dealing with modules, asyncio for concurrency, json for parsing, and pandas for tabular storage, alongside Crawl4AI’s necessities: AsyncWebCrawler to drive the crawl, CrawlerRunConfig and HTTPCrawlerConfig to configure extraction and HTTP settings, AsyncHTTPCrawlerStrategy for a browser‑free HTTP backend, and JsonCssExtractionStrategy to map CSS selectors into structured JSON.
http_cfg = HTTPCrawlerConfig(
technique="GET",
headers={
"Person-Agent": "crawl4ai-bot/1.0",
"Settle for-Encoding": "gzip, deflate"
},
follow_redirects=True,
verify_ssl=True
)
crawler_strategy = AsyncHTTPCrawlerStrategy(browser_config=http_cfg)
Right here, we instantiate an HTTPCrawlerConfig to outline our HTTP crawler’s conduct, utilizing a GET request with a customized Person-Agent, gzip/deflate encoding solely, automated redirects, and SSL verification. We then plug that into AsyncHTTPCrawlerStrategy, permitting Crawl4AI to drive the crawl by way of pure HTTP calls fairly than a full browser.
schema = {
"title": "Quotes",
"baseSelector": "div.quote",
"fields": [
{"name": "quote", "selector": "span.text", "type": "text"},
{"name": "author", "selector": "small.author", "type": "text"},
{"name": "tags", "selector": "div.tags a.tag", "type": "text"}
]
}
extraction_strategy = JsonCssExtractionStrategy(schema, verbose=False)
run_cfg = CrawlerRunConfig(extraction_strategy=extraction_strategy)
We outline a JSON‑CSS extraction schema concentrating on every quote block (div.quote) and its little one components (span.textual content, small.creator, div.tags a.tag), then initializes a JsonCssExtractionStrategy with that schema, and wraps it in a CrawlerRunConfig so Crawl4AI is aware of precisely what structured knowledge to tug on every request.
async def crawl_quotes_http(max_pages=5):
all_items = []
async with AsyncWebCrawler(crawler_strategy=crawler_strategy) as crawler:
for p in vary(1, max_pages+1):
url = f"https://quotes.toscrape.com/web page/{p}/"
attempt:
res = await crawler.arun(url=url, config=run_cfg)
besides Exception as e:
print(f"❌ Web page {p} failed outright: {e}")
proceed
if not res.extracted_content:
print(f"❌ Web page {p} returned no content material, skipping")
proceed
attempt:
objects = json.hundreds(res.extracted_content)
besides Exception as e:
print(f"❌ Web page {p} JSON‑parse error: {e}")
proceed
print(f"✅ Web page {p}: {len(objects)} quotes")
all_items.prolong(objects)
return pd.DataFrame(all_items)Now, this asynchronous operate orchestrates the HTTP‑solely crawl: it spins up an AsyncWebCrawler with our AsyncHTTPCrawlerStrategy, iterates by means of every web page URL, and safely awaits crawler.arun(), handles any request or JSON parsing errors and collects the extracted quote information right into a single pandas DataFrame for downstream evaluation.
df = asyncio.get_event_loop().run_until_complete(crawl_quotes_http(max_pages=3))
df.head()
Lastly, we kick off the crawl_quotes_http coroutine on Colab’s current asyncio loop, fetching three pages of quotes, after which show the primary few rows of the ensuing pandas DataFrame to confirm that our crawler returned structured knowledge as anticipated.
In conclusion, by combining Google Colab’s zero-config atmosphere with Python’s asynchronous ecosystem and Crawl4AI’s versatile crawling methods, we’ve got now developed a totally automated pipeline for scraping and structuring net knowledge in minutes. Whether or not you might want to spin up a fast dataset of quotes, construct a refreshable information‑article archive, or energy a RAG workflow, Crawl4AI’s mix of httpx, asyncio, JsonCssExtractionStrategy, and AsyncHTTPCrawlerStrategy delivers each simplicity and scalability. Past pure HTTP crawls, you possibly can immediately pivot to Playwright‑pushed browser automation with out rewriting your extraction logic, underscoring why Crawl4AI stands out because the go‑to framework for contemporary, manufacturing‑prepared net knowledge extraction.
Right here is the Colab Notebook. Additionally, don’t overlook to observe us on Twitter and be a part of our Telegram Channel and LinkedIn Group. Don’t Overlook to hitch our 90k+ ML SubReddit.
Nikhil is an intern advisor at Marktechpost. He’s pursuing an built-in twin diploma in Supplies on the Indian Institute of Know-how, Kharagpur. Nikhil is an AI/ML fanatic who’s at all times researching purposes in fields like biomaterials and biomedical science. With a robust background in Materials Science, he’s exploring new developments and creating alternatives to contribute.
