Crawl a site

Collect every page under a path, for docs ingestion or a content audit. About 3 minutes.

You need:

• An API key in PYROSYNC_API_KEY and the deployment URL in PYROSYNC_URL (for example https://app.example.com).

1. Start the crawl

POST /api/v1/crawl

CRAWL_ID=$(curl -sS -X POST "$PYROSYNC_URL/api/v1/crawl" \
  -H "X-API-Key: $PYROSYNC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "url": "https://example.com/docs",
  "limit": 20,
  "maxDepth": 2
}' | jq -r '.data.id')
echo "$CRAWL_ID"

2. Poll until it finishes

GET /api/v1/crawl/{id}

curl -sS -X GET "$PYROSYNC_URL/api/v1/crawl/$CRAWL_ID" \
  -H "X-API-Key: $PYROSYNC_API_KEY"

Repeat every few seconds until data.status is completed. GET /api/v1/crawl/{id}/events streams progress instead.

Result: Up to 20 pages, each with Markdown and metadata.

Cost: One credit per crawled page, capped by limit.