Purpose-built web crawler for feeding LLMs - it strips a page down to clean markdown instead of leaving you to parse raw HTML yourself.
Where crawl4ai fits, based on its real tags and topics.
git clone https://github.com/unclecode/crawl4ai.gitSee the repository's README for language-specific setup steps.
🚀🤖 Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper.
[](https://github.com/unclecode/crawl4ai/stargazers) [](https://github.com/unclecode/crawl4ai/network/members)
[](https://badge.fury.io/py/crawl4ai) [](https://pypi.org/project/crawl4ai/) [](https://pepy.tech/project/crawl4ai) [](https://github.com/sponsors/unclecode)
Category: 🤖 AI Agents
Difficulty: Intermediate
“The default choice for production React today - server components, file-based routing, and first-class deployment story make it the fastest path from idea to shipped app.”
The Postgres development platform. Supabase gives you a dedicated Postgres database to build your web, mobile, and AI applications.
“A genuinely open-source Firebase alternative - real Postgres underneath, with auth, storage, and realtime built in rather than a proprietary database.”
“Utility-first CSS that's become the default styling approach for new projects - fast to write, easy to keep consistent, no dead CSS to prune later.”
Find vulnerabilities, misconfigurations, secrets, SBOM in containers, Kubernetes, code repositories, clouds and more
“A fast, all-in-one vulnerability scanner - containers, filesystems, git repos, and Terraform/IaC misconfigurations all covered by one CLI tool.”
“The most widely adopted framework for chaining LLM calls, tools, and retrieval into real applications - a near-default starting point for RAG and agent projects.”
The AI Toolkit for TypeScript. From the creators of Next.js, the AI SDK is a free open-source library for building AI-powered applications and agents
“The cleanest way to wire streaming LLM responses into a React/Next.js UI - handles the hard parts (streaming, tool calls, edge runtimes) so you don't have to.”
“The official, minimal-overhead client for OpenAI's API - the right starting point before reaching for a bigger framework.”
“The standard library for working with open-weight models - thousands of pretrained models are one line of code away.”
“One of the earliest serious multi-agent orchestration frameworks - useful once a single agent/chain isn't enough for the problem.”
“A lighter-weight take on multi-agent orchestration - agents get explicit roles and goals, which reads more naturally than raw chain composition.”
“The industry-standard container orchestrator - not always necessary at small scale, but the default assumption for any serious cloud-native platform team.”
“The most capable open-source relational database available - strong standards compliance, extensions for nearly everything (including full-text search and vectors), and a default choice for good reason.”
“Unifies dozens of LLM providers behind a single API, making model switching almost effortless in production.”
“The modern default frontend build tool - native ESM in dev means a dev server that starts and reloads almost instantly, even on large apps.”
“The open-source engine behind Docker - understanding containers here pays off no matter which orchestrator sits on top later.”
“The standard tool for infrastructure-as-code across every major cloud provider - one declarative language instead of provider-specific consoles or scripts.”
