Bonus 1 — Local News RAG: Pointing the Pattern at Live Data

Same skeleton, live target

You just built a support bot over static documents. This bonus proves the pattern is bigger than one project: we’ll aim the exact same RAG skeleton at live news, building an assistant that answers questions about today’s local news with citations back to the source articles.

Almost everything carries over unchanged — the chunker from Episode 2, the embedder from Episode 3, the normalize helper and dot-product function from Episode 4, the hybrid retrieval from Episode 7, the grounding prompt from Episode 6. That’s the whole point of learning RAG properly: the skeleton is reusable, and only the edges change.

But live data brings three genuinely new problems that static docs never had:

1. Freshness — the knowledge base changes constantly; yesterday’s index is stale.
2. Duplication — the same story arrives from multiple outlets and multiple fetches; embed it twice and your bot repeats itself.
3. Recency — “what happened today?” needs a sense of time, which our support docs never carried.

This post handles the first two (ingestion). Bonus 2 handles the third (recency-aware answers). Less theory this time — you own the theory — more building.

Step 1 — Choosing a news source (and not marrying it)

We’ll use an open news API. This walkthrough uses GNews because its free tier is simple and generous enough to learn on (100 requests/day, JSON responses), but the smart move is to not hard-wire any provider. News APIs come and go, rate limits change, and you may want to swap GNews for NewsData.io, NewsAPI, or even plain RSS feeds later.

So we isolate the provider behind a small adapter — one function that turns that provider’s response into our standard shape. Swap providers later and only this function changes.

💡 What’s an “adapter”? A thin translation layer. Every news API returns slightly different JSON. Instead of letting those differences leak through your whole codebase, you convert each provider’s format into one internal shape at the door. The rest of your code only ever sees the clean, standard shape.

Getting your free GNews API key (takes about a minute, no credit card):

1. Register at gnews.io/register and verify your email.
2. Open your dashboard — your unique API key is shown there.
3. Copy it into .env below.

The free tier gives you 100 requests/day, which is plenty for building and testing. The full endpoint and parameter reference lives in the official GNews docs, and authentication specifics are here.

Add your key to .env:

GNEWS_API_KEY=your-gnews-key-here

<?php
// news-source.php — fetch news and normalize it to our standard shape
require __DIR__ . '/config.php';

function httpGetJson(string $url): array {
    $ch = curl_init($url);
    curl_setopt_array($ch, [
        CURLOPT_RETURNTRANSFER => true,
        CURLOPT_TIMEOUT        => 15,
    ]);
    $body = curl_exec($ch);
    if ($body === false) {
        throw new RuntimeException('Fetch failed: ' . curl_error($ch));
    }
    curl_close($ch);
    return json_decode($body, true) ?? [];
}

// Fetch raw articles from GNews for a topic/place
function fetchNews(string $query, string $country = 'in', int $max = 10): array {
    $url = 'https://gnews.io/api/v4/search?' . http_build_query([
        'q'       => $query,
        'country' => $country,     // e.g. 'in', 'us', 'gb' — your locale
        'lang'    => 'en',
        'max'     => $max,
        'apikey'  => env('GNEWS_API_KEY'),
    ]);

    $data = httpGetJson($url);
    return array_map('normalizeArticle', $data['articles'] ?? []);
}

// THE ADAPTER: GNews shape → our standard shape.
// Swap providers by rewriting only this function.
function normalizeArticle(array $a): array {
    return [
        'url'          => $a['url']                 ?? '',
        'title'        => trim($a['title']          ?? ''),
        'description'  => trim($a['description']    ?? ''),
        'content'      => trim($a['content']        ?? ''),
        'source_name'  => $a['source']['name']      ?? 'Unknown',
        'published_at' => isset($a['publishedAt'])
            ? date('Y-m-d H:i:s', strtotime($a['publishedAt']))  // ISO → MySQL DATETIME
            : null,
    ];
}

⚠️ Free-tier reality: on GNews’s free plan the content field is truncated — you get the title and a description-length snippet, not the full article (full text needs a paid expand=content). That’s fine for learning, and it shapes a real design decision: with short text per article, most articles become a single chunk. We’ll still run them through the chunker so the code handles long articles too if you upgrade or switch to RSS full-text.

Step 2 — A schema built for live data

Our support-bot table didn’t need timestamps or a dedup key. This one does. Notice what’s new: published_at (for recency), fetched_at (for cleanup), and a unique key on the URL so the same article can’t be stored twice.

CREATE TABLE news_chunks (
    id           INT AUTO_INCREMENT PRIMARY KEY,
    article_url  VARCHAR(700),
    title        VARCHAR(500),
    source_name  VARCHAR(255),
    published_at DATETIME NULL,
    chunk_index  INT,
    content      TEXT,
    embedding    JSON,
    fetched_at   TIMESTAMP DEFAULT CURRENT_TIMESTAMP,

    UNIQUE KEY uniq_article_chunk (article_url(255), chunk_index),
    FULLTEXT INDEX idx_ft_content (content),
    INDEX idx_published (published_at)
) CHARACTER SET utf8mb4;

That UNIQUE KEY is doing real work: paired with INSERT IGNORE, it makes re-fetching safe. Run the ingester ten times and each unique article is still stored exactly once — the database enforces it, so our PHP doesn’t have to.

Step 3 — Ingesting the news

Now the pipeline. It looks familiar because it is familiar — the same load → chunk → embed → store flow from Episode 2 and 3, with dedup baked in. We reuse chunker.php, embed.php, and vector.php untouched.

<?php
// ingest-news.php — fetch, chunk, embed, and store news (dedup-safe)
require __DIR__ . '/db.php';
require __DIR__ . '/embed.php';
require __DIR__ . '/vector.php';
require __DIR__ . '/chunker.php';       // Episode 2, unchanged
require __DIR__ . '/news-source.php';

$topics  = $argv[1] ?? 'Mumbai';        // your city / topic
$articles = fetchNews($topics, 'in', 10);

$pdo  = db();
$stmt = $pdo->prepare(
    'INSERT IGNORE INTO news_chunks
       (article_url, title, source_name, published_at, chunk_index, content, embedding)
     VALUES (?, ?, ?, ?, ?, ?, ?)'
);

$stored = $skipped = 0;

foreach ($articles as $a) {
    if ($a['url'] === '' || $a['title'] === '') {
        continue;
    }

    // Combine the fields we actually have into one text blob
    $text = $a['title'];
    if ($a['description'] !== '') $text .= ". " . $a['description'];
    if ($a['content'] !== '')     $text .= "\n\n" . $a['content'];

    $chunks = chunkText($text, 800, 150);

    foreach ($chunks as $i => $chunk) {
        $embedding = normalize(getEmbedding($chunk));
        $ok = $stmt->execute([
            $a['url'], $a['title'], $a['source_name'],
            $a['published_at'], $i, $chunk, json_encode($embedding),
        ]);
        // rowCount() is 0 when INSERT IGNORE skipped a duplicate
        $stmt->rowCount() > 0 ? $stored++ : $skipped++;
    }
}

echo "Stored {$stored} new chunks, skipped {$skipped} duplicates.\n";

Run it:

php bonus-01/ingest-news.php "Mumbai"

Stored 14 new chunks, skipped 0 duplicates.

Run it again immediately:

Stored 0 new chunks, skipped 14 duplicates.

That second run is the dedup guard proving itself — nothing embedded twice, no wasted API cost, no repetitive answers. Exactly what live data needs.

⚠️ A subtler duplication problem: the same story from two outlets has two different URLs, so the URL key won’t catch it. For a learning project that’s acceptable. If it bothers you, near-duplicate detection using the embeddings themselves (cosine similarity above ~0.95 = probably the same story) is the natural next step — a nice exercise once you finish the series.

Step 4 — Retrieval over news

Retrieval is almost a copy of Episode 7’s hybrid search — pointed at news_chunks and carrying the extra metadata (source, URL, published date) that news answers need for citation. Here’s the news-flavored version:

<?php
// retrieve-news.php — hybrid retrieval over the news index
require __DIR__ . '/db.php';
require __DIR__ . '/embed.php';
require __DIR__ . '/vector.php';

function retrieveNews(string $question, int $topK = 5, float $minScore = 0.27): array {
    $queryVec = normalize(getEmbedding($question));
    $pool     = 20;

    // Vector candidates
    $v = db()->prepare(
        'SELECT id, article_url, title, source_name, published_at, content,
                dot_product(embedding, CAST(:qvec AS JSON)) AS score
           FROM news_chunks
       ORDER BY score DESC
          LIMIT :k'
    );
    $v->bindValue(':qvec', json_encode($queryVec));
    $v->bindValue(':k', $pool, PDO::PARAM_INT);
    $v->execute();
    $vec = $v->fetchAll();

    // Keyword candidates (FULLTEXT)
    $kw = db()->prepare(
        'SELECT id, article_url, title, source_name, published_at, content,
                MATCH(content) AGAINST(:q IN NATURAL LANGUAGE MODE) AS score
           FROM news_chunks
          WHERE MATCH(content) AGAINST(:q2 IN NATURAL LANGUAGE MODE)
       ORDER BY score DESC
          LIMIT :k'
    );
    $kw->bindValue(':q', $question);
    $kw->bindValue(':q2', $question);
    $kw->bindValue(':k', $pool, PDO::PARAM_INT);
    $kw->execute();
    $key = $kw->fetchAll();

    // "Nothing relevant" gate, same idea as Episode 5
    if (($vec[0]['score'] ?? 0) < $minScore && empty($key)) {
        return [];
    }

    // Reuse Reciprocal Rank Fusion from Episode 7
    $merged = reciprocalRankFusion([$vec, $key]);
    return array_slice($merged, 0, $topK);
}

reciprocalRankFusion() is the same function from Episode 7 — pull it into a shared file and both projects use it. Quick test:

<?php
require __DIR__ . '/retrieve-news.php';
require __DIR__ . '/hybrid.php';   // for reciprocalRankFusion()

foreach (retrieveNews($argv[1] ?? 'Mumbai traffic') as $i => $r) {
    printf("#%d  [%s | %s]\n%s\n\n",
        $i + 1,
        $r['source_name'],
        $r['published_at'],
        mb_substr($r['title'] . ' — ' . $r['content'], 0, 140) . '…'
    );
}

php bonus-01/test-news.php "what's happening with local trains?"

You’ll get back the most relevant recent articles, each tagged with its outlet and publish time — the raw material Bonus 2 turns into a cited, recency-aware answer.

Step 5 — Keeping it fresh

Static docs are ingested once. News is never “done.” Two operational habits make this work:

Re-fetch on a schedule. A cron job runs ingest-news.php every few hours. Thanks to INSERT IGNORE, it only ever adds genuinely new articles — cheap and safe to run often.

# crontab: refresh local news every 3 hours
0 */3 * * * php /path/to/bonus-01/ingest-news.php "Mumbai" >> /var/log/news-rag.log 2>&1

Expire the old. Yesterday’s news shouldn’t answer “what’s happening today?” A tiny cleanup keeps the index current:

DELETE FROM news_chunks WHERE published_at < NOW() - INTERVAL 7 DAY;

💡 Design shift worth naming: a support-doc index is a library — you curate it and it sits still. A news index is a river — it flows, and your job is to keep dipping fresh water in and letting old water out. Same retrieval machinery, opposite lifecycle.

Try it yourself

1. Localize it. Change the topic and country code to your own city and region. Ingest, then ask about something you know is in today’s news.
2. Prove dedup works. Run the ingester three times back to back. Only the first should store anything.
3. Swap the source. Sign up for NewsData.io (or grab an RSS feed) and rewrite only normalizeArticle() and fetchNews(). Everything downstream should keep working — that’s the adapter earning its keep.
4. Watch freshness. Ingest today, wait a day, ingest again, and run the expiry query. Confirm old chunks leave and new ones arrive.

Recap & what’s next

• The RAG skeleton is fully reusable — chunker, embedder, vector math, hybrid retrieval all carried over untouched from the core series.
• Live data adds three challenges: freshness, duplication, and recency.
• An adapter (normalizeArticle) isolates the news provider so you can swap APIs without touching the rest of the code.
• A unique URL key + INSERT IGNORE makes re-fetching safe and dedup automatic — the database enforces it for you.
• New metadata (published_at, source_name, article_url) is what will power recency and citations next.
• A news index is a river, not a library: schedule re-fetches, expire the old.

We can now fetch, dedup, and retrieve today’s news. But retrieval just hands us articles — we haven’t generated an answer yet, and news answers have their own quirks: they must respect time (“today” vs “last week”), synthesize across multiple articles, and cite several outlets at once. In Bonus 2 — Prompts & Polish, we’ll write the news-specific generation: a recency-aware grounding prompt, multi-article citations with links and timestamps, and the edge cases that make a news assistant feel trustworthy. Bring your bot.php — we’re wiring the last piece.

GNews links: Register / get API key · Dashboard · Documentation · Authentication · JSON response reference. News-API comparison: Best News APIs, 2026.