Skip to main content
POST
/
ai-agents
/
agent-builder
/
knowledge-base
/
website
/
individual-page
Scrape Single Page
curl --request POST \
  --url https://{appid}.api-{region}.cometchat.io/v3/ai-agents/agent-builder/knowledge-base/website/individual-page \
  --header 'Content-Type: application/json' \
  --header 'apikey: <api-key>' \
  --data '
{
  "url": "https://docs.example.com/getting-started",
  "timeout": 30000,
  "format": "markdown"
}
'
{
  "success": true,
  "message": "Page scraped successfully",
  "data": {
    "url": "https://docs.example.com/getting-started",
    "title": "Getting Started Guide",
    "description": "Learn how to get started with our platform",
    "content": "# Getting Started\n\nWelcome to our platform...",
    "metadata": {
      "og": {},
      "twitter": {},
      "canonical": "<string>",
      "language": "<string>"
    },
    "scraper": "Firecrawl",
    "crawlDuration": 2500,
    "contentLength": 1024,
    "timestamp": "2025-12-05T17:30:00Z"
  }
}

Authorizations

​
apikey
string
header
required

API Key with fullAccess scope(i.e. Rest API Key from the Dashboard).

Body

application/json

Single page scraping configuration

​
url
string
required

Target page URL to crawl and extract content from

Example:

"https://docs.example.com/getting-started"

​
timeout
number
default:30000

Request timeout in milliseconds

Required range: 5000 <= x <= 120000
Example:

30000

​
crawlerType
string
default:firecrawl

Crawler service to use for page crawling (firecrawl, puppeteer, etc.)

Example:

"firecrawl"

​
format
enum<string>
default:markdown

Extract content in specific format

Available options:
markdown,
html,
text
Example:

"markdown"

Response

Page scraped successfully

​
success
boolean
Example:

true

​
message
string
Example:

"Page scraped successfully"

​
data
object