Loading

Firecrawl connector

The Firecrawl connector uses the Firecrawl REST API v2 to scrape web pages, search the web, map site URLs, and crawl websites. It supports Bearer token (API key) authentication.

You can create connectors in Stack Management > Connectors.

Firecrawl connectors have the following configuration properties:

API Key (Bearer token)
Your Firecrawl API key for authentication. The connector sends it in the Authorization: Bearer <key> header to https://api.firecrawl.dev.

You can test connectors when you create or edit the connector in Kibana.

The Firecrawl connector has the following actions:

scrape

Scrape a single URL and extract content (for example, Markdown or HTML).

  • url (required): The URL to scrape.
  • formats (optional): Output formats: markdown, html, links, screenshot, extract. Defaults to Markdown.
  • onlyMainContent (optional): Return only main content, excluding navigation and footer elements. Defaults to true.
  • waitFor (optional): Delay in milliseconds before fetching (useful for JavaScript-rendered pages).
search

Search the web and optionally get full content from results.

  • query (required): Search query string.
  • limit (optional): Maximum number of results (1 to 100). Defaults to 5.
map

Map a website to discover indexed URLs.

  • url (required): Base URL to map.
  • search (optional): Search term to filter or order URLs by relevance.
  • limit (optional): Maximum URLs to return. Defaults to 5000.
  • includeSubdomains (optional): Include subdomains. Defaults to true.
  • ignoreQueryParameters (optional): Deduplicate by path only. Defaults to true.
crawl

Start an asynchronous crawl of a website. Returns a job ID. Use getCrawlStatus to poll for results.

  • url (required): Base URL to start crawling from.
  • limit (optional): Maximum pages to crawl. Defaults to 100.
  • maxDiscoveryDepth (optional): Maximum discovery depth.
  • allowExternalLinks (optional): Follow external links. Defaults to false.
getCrawlStatus

Get the status and results of a crawl job.

  • id (required): The crawl job ID (UUID) returned from the crawl action.

Use the Action configuration settings to customize connector networking, such as proxies, certificates, or TLS settings. You can set configurations that apply to all your connectors or use xpack.actions.customHostSettings to set per-host configurations.

To use the Firecrawl connector:

  1. Sign up at Firecrawl or log in to your account.
  2. Go to API keys in the Firecrawl dashboard.
  3. Create an API key or copy an existing one (for example, fc-...).
  4. Enter the API key in the API Key field when configuring the Firecrawl connector in Kibana.

Rate limits and billing depend on your Firecrawl plan. Refer to Firecrawl billing for details.