Research Methods

The Resilience Scanner employs a hybrid research approach combining systematic document analysis with autonomous AI-driven web research. Initial solution identification comes from manual review of municipal climate plans, while in-depth research and verification leverages a sophisticated autonomous agent architecture designed to maximize factual accuracy while minimizing hallucination risk.

Identifying Solutions

We systematically reviewed climate adaptation and resilience plans from over 100 global cities. Documents were sourced from official municipal websites, climate action networks (C40, ICLEI, 100 Resilient Cities), and international urban resilience initiatives. Large language models (GPT-4 and Claude) were employed to identify and extract technology-enabled adaptation solutions from municipal planning documents, with each solution undergoing multiple extraction passes to ensure accuracy and completeness. Our focus is on technology-enabled interventions rather than policy-only measures, capturing specific implementation details, local context, stakeholder information, and measurable outcomes.

Deep Research

For each identified solution, an autonomous research agent conducts comprehensive web research to gather authoritative information, verify claims, and generate detailed case studies. This approach allows us to go beyond what appears in planning documents to find implementation details, performance data, costs, timelines, and stakeholder information from diverse web sources that have been published since the original plan where the solution was proposed or documented.

Agent Design Philosophy

Our research agent employs a supervisor/orchestrator pattern where a large language model acts as a research director rather than content generator. The agent uses structured function calling to invoke specialized tools, maintaining transparency and control throughout the research process. Rather than generating content from its training data, the agent plans research strategies, selects appropriate tools, evaluates information quality, and iteratively refines its approach based on real-time assessment of coverage gaps.

The agent operates through an iterative workflow of search, extraction, validation, and gap analysis. Each research session typically involves 8-12 iterations where the agent searches for sources, scrapes promising web pages and PDFs, extracts factual claims with supporting evidence, validates those claims against quality thresholds, identifies missing information categories, and adjusts its search strategy accordingly. This dynamic planning approach allows the agent to persist through challenges and adapt to varying levels of documentation availability across different solutions.

The agent maintains a structured research state throughout execution, tracking validated claims, scraped URLs, search queries, citation mappings, and knowledge coverage across information categories. This state drives decision-making and prevents duplication while building toward comprehensive coverage of each solution.

Preventing Hallucinations

The architecture implements multiple overlapping validation layers designed to ensure every factual claim in generated research is grounded in authoritative web sources rather than LLM training data.

Source-Grounded Claims Architecture. The system never generates factual content from LLM parametric knowledge. Every factual statement must be extracted from a specific web source, include a supporting text snippet from that source, and link to the source URL for verification. Claims are extracted as structured data objects that enforce the requirement that facts come from documents, not model training. This architectural constraint fundamentally prevents the most common form of hallucination where models generate plausible-sounding but unverified statements.

Three-Phase Claim Validation. Each extracted claim undergoes comprehensive validation before inclusion in final research. First, the system verifies that the supporting snippet actually exists in the source document using fuzzy string matching, immediately rejecting claims with fabricated or misattributed evidence. Second, an LLM evaluates claim accuracy by assessing how well the claim is supported by the snippet, checking for specificity versus vagueness, evaluating relevance to the particular solution being researched, and identifying potential red flags like speculation or outdated information. Third, claims are scored quantitatively across multiple dimensions with deterministic thresholds applied—only claims meeting minimum quality standards are included in research outputs. This validation typically results in 60-75% of extracted claims passing filters, with rejected claims logged alongside detailed explanations. The selective approach prioritizes accuracy over volume, ensuring that generated research contains only well-supported factual statements.

Multi-Source Triangulation. Research quality requirements mandate minimum source diversity. Each solution requires at least three distinct authoritative sources, with targets of 5-10 validated claims from diverse sources. The agent prioritizes government, academic, and organizational domains when selecting sources to scrape and extract from. Coverage analysis tracks whether claims span critical information categories including financial details, technical specifications, implementation timelines, performance metrics, stakeholder information, geographic context, and comparative examples. This breadth requirement prevents narrow single-source narratives and encourages comprehensive research.

Transparent Citations. Every claim included in final narratives includes an inline hyperlinked citation linking directly to the source document. Citations are deduplicated so each unique source receives a single numbered reference regardless of how many claims are extracted from it. This transparency allows readers to verify claims against original sources, assess source authority and credibility, and identify potential biases or limitations. All generated research is clearly marked as AI-assisted with proper attribution to the specific models used.

Limitations and Future Considerations

There are several limitations with our current methods that will be the focus of future design and engineering work.

Language Constraints: Current analysis focuses on English-language documents and web sources, which may underrepresent solutions from non-English speaking regions.

Web Source Availability: Research quality depends on availability of public web documentation. Solutions with limited online documentation receive shorter narratives based on available information.

Temporal Coverage: The database reflects solutions as documented at the time of research execution, with continuous updates to capture recent developments.

Source Authority Bias: Prioritization of government, academic, and organizational domains may underrepresent solutions documented primarily in news media or technical blogs, though these sources are included when more authoritative sources are unavailable.

LLM Capabilities: While the multi-layer validation architecture substantially reduces hallucination risk, LLM-based systems cannot achieve perfect accuracy. All generated research should be considered AI-assisted analysis rather than definitive fact.

Last Updated: October 2025