Worker

Curated Competitor-News Newsletter from Live Crawls

A scheduled worker that crawls competitor sites and news sources with Firecrawl, has an LLM cluster and summarize the fresh items grounded in the scraped text, and assembles a curated newsletter draft with source links.

What This Builds

This recipe builds a newsletter worker that keeps an audience current on an industry without manual reading. On a schedule, Firecrawl crawls a set of competitor sites and news pages, returns clean markdown plus images, and an LLM summarizes and groups the fresh items into a curated newsletter. Because every section is grounded in scraped text — and Firecrawl preserves source URLs and timestamps — the draft is verifiable, not invented. Firecrawl’s content-generation use case explicitly calls out “Newsletters: curated updates from industry and competitor news.”

The Stack

  • Firecrawl crawls and batch-scrapes the target URLs, returning markdown/JSON/HTML, images, and screenshots, each tagged with source URL and timestamp for verification.
  • Anthropic | Claude clusters the scraped items by theme and writes concise summaries grounded in the extracted text.
  • Trigger.dev runs the crawl-summarize-assemble job on a recurring schedule with retries and observability.
  • Resend delivers the assembled newsletter draft to a review inbox or list.

Step-by-Step Outline

  1. List the competitor domains and news pages you want to track.
  2. In a scheduled Trigger.dev task, call Firecrawl’s crawl/batch-scrape API to pull fresh pages as clean markdown plus images.
  3. Filter to items newer than the last run using Firecrawl’s returned timestamps.
  4. Send the scraped text to Claude with a prompt that clusters items by topic and writes a short summary for each, keeping source links.
  5. Assemble the summaries into an HTML newsletter, enriched with extracted images/screenshots.
  6. Email the draft via Resend to an editor for a final pass before it goes to the list.

Why This Shape Works

Grounding every summary in freshly scraped source text — with URLs and timestamps attached — is what separates a trustworthy curated newsletter from AI guesswork. Firecrawl handles the messy extraction so the LLM only reasons over clean input, and the scheduled Trigger.dev wrapper makes the whole thing a hands-off worker that produces a reviewable draft each cycle.

Source

Firecrawl Docs, “Content Generation” use case (newsletters from industry and competitor news): https://docs.firecrawl.dev/use-cases/content-generation