Crawl any website into your Dify Knowledge Base. Web data infrastructure for developers and enterprises, used by the fastest-growing AI companies.
Olostep is web data infrastructure for AI agents and enterprise AI workflows. Its APIs provide live web search, scraping and crawling to extract, clean and structure data from the web.
This data source plugin crawls any website into your Dify Knowledge Base as clean, LLM-ready markdown. To use Olostep in agents and workflows, install the Olostep tool plugin.
Source repository: https://github.com/olostep-api/olostep-dify-datasource
From extracting data from a handful of URLs to crawling hundreds of millions of pages, Olostep gives developers and AI companies reliable, scalable and highly cost-effective access to clean, structured web data.
Enter a start URL and choose your options:
Each page is returned with its URL, markdown content, title and description, ready for chunking in your Knowledge Pipeline.
The plugin connects only to the Olostep API at over HTTPS. Your Dify instance needs outbound network access to that host. Rate limits and server errors are retried automatically.
The plugin needs an Olostep API key. It is stored by Dify as a secret and sent to the Olostep API in the header.
Last checked 11/03/2026 05:56 AM · version 0.1.0