Search Layer

Overview

TerraGuard's search layer discovers news articles, situation reports, and official advisories about disaster events. It is an API-based layer: the Backend API calls commercial search APIs directly. There is no self-hosted search engine and no Tor proxy in the request path.

The layer lives inside the Backend API as a shared module (app/common/search_providers.py) and exposes two entry points — search_web() and search_news() — used by event knowledge discovery, the news crawler, situation-report generation, system-health checks, and the agentic web-search tool.

Architecture

The fallback is error-triggered only: Brave is called when Serper raises an error (missing key, network failure, upstream 5xx). A successful-but-empty Serper response does not trigger fallback.

Providers

Serper.dev proxies Google's web and news indexes through a single API call, which removed the need for the SearXNG meta-search engine and the Tor anonymity layer that previously guarded against IP rate-limiting.

Unified result model

Both providers normalize into one SearchResult shape so consumers never branch on provider:

@dataclass
class SearchResult:
    url: str
    title: str
    snippet: str
    source: str        # "serper" | "brave"
    search_type: str   # "web" | "news" | "siterep"
    published_date: Optional[str] = None
    position: Optional[int] = None

Freshness

A single UnifiedFreshness object maps one freshness bucket to each provider's own parameter, so callers pick a window once:

age_to_freshness(anchor) picks the smallest bucket containing now − anchor, so a freshly detected event searches past_day while an older one widens automatically.

Entry points

from app.common.search_providers import search_web, search_news, age_to_freshness

# Web search — Serper primary, Brave fallback on error
results = await search_web("earthquake Turkey 2026", freshness=age_to_freshness(event.detected_at), limit=100)

# News search
news = await search_news("Cyclone Mocha landfall", limit=50)

Both return a flat list[SearchResult]. When Serper errors, the orchestrator appends Brave results to whatever Serper returned and logs the fallback.

Configuration

Both live in the Backend API's environment (app/config.py). If SERPER_API_KEY is unset, every web/news call falls straight through to Brave; if both are unset, search returns an empty result set with a logged error rather than raising.

What was removed

The previous search stack — a standalone tg-search-api Go service backed by SearXNG (meta-searching Google/Bing/DuckDuckGo via a Tor SOCKS proxy) with Brave as a fallback — has been decommissioned and removed from the system: the SearXNG and Tor containers are gone from the dev and production topology, and the backend no longer makes any call to a search service.

It was removed because:

• Serper.dev returns Google-quality web + news results through a single authenticated API, removing the operational burden of self-hosting SearXNG.
• The Tor anonymity layer (needed to dodge IP rate-limiting on scraped engines) is unnecessary against a paid API.
• Collapsing search into a backend module removed a network hop, two containers (SearXNG + Tor), and the SearXNG/Tor/residential-proxy failure modes.

The result is a two-provider layer — Serper primary, Brave fallback — with no self-hosted search infrastructure.