Lakshya

Software Engineering · Chapter 30 of 31

APIs, webhooks and MCP

Three ways software talks to software, what each is genuinely for, and a worked Confluence chatbot using two of them.

11 min read2 diagramsAll 31 chapters

These three come up constantly and get muddled, usually because people compare them as competing technologies. They are not alternatives. They answer different questions, and once you see which, the choice stops being a debate.

WHO STARTS THE CONVERSATION? THAT IS THE WHOLE DISTINCTION. API — YOU ASK. You start it, you wait, you get an answer. YOUR CODE “give me page 12345” the page CONFLUENCE Good for: getting something now. Bad for: knowing when something CHANGED. WEBHOOK — YOU ARE TOLD. They start it. It is an API call in reverse. YOUR SERVER “page 12345 just changed” CONFLUENCE Good for: reacting the moment it happens. You must run a public endpoint to receive it. MCP — THE MODEL ASKS. You hand it tools; it decides which to call, and when. THE MODEL “search_confluence(‘refund policy’)” MCP SERVER which calls the ordinary Confluence API underneath. MCP does not replace the API. It is a STANDARD WRAPPER that makes an API describable to a model. Without it you hand-write a tool definition per API, per agent framework. With it, one server works with every client that speaks MCP.
Read the left column: you ask, you are told, the model asks. Everything else about these three follows from that one difference.

API — you ask

An API (Application Programming Interface) is a way for one program to ask another for something. You send a request, you wait, you get an answer. Everything on the internet is built on this.

  • REST is the common style: a URL identifies a thing, and the HTTP method says what to do with it. GET /pages/12345 reads it; POST creates; PUT replaces; DELETE removes.
  • GraphQL is the main alternative: one endpoint, and the caller specifies exactly which fields it wants. Useful when clients need very different slices of the same data; more work to secure and cache.
  • An OpenAPI specification is a machine-readable description of a REST API — every endpoint, parameter and response shape. Remember this one; it is what MCP is doing for models.

What an API cannot do is tell you something happened. To notice a change you must ask again — polling — and polling is a bad trade in both directions: poll often and you waste almost every request discovering nothing changed; poll rarely and you find out late. That gap is exactly what webhooks exist to fill.

Webhook — you are told

A webhook is an API call in reverse. You register a URL with the other system; when the event happens, it calls you. No polling, no delay.

The mental shift is that you are now running a server, with a publicly reachable endpoint, that receives requests from outside on a schedule you do not control. That brings four requirements, and skipping any of them is how webhook integrations fail:

  • Verify the signature. Your endpoint is a public URL — anyone can post to it. Real senders sign each payload with a shared secret; recompute the signature and reject on mismatch. An unverified webhook endpoint is an unauthenticated write into your system, and it is startlingly common.
  • Respond immediately, then do the work. Acknowledge with a 200 in milliseconds and queue the actual processing. Senders time out, and a slow endpoint gets retried — or disabled.
  • Be idempotent. Every serious sender retries on failure, and retries sometimes arrive for messages you already processed. Deduplicate on the event ID. ‘At least once’ delivery is the norm, not an edge case.
  • Expect them out of order, and to be dropped. Treat a webhook as a hint that something changed, not as the data itself — then fetch the current state via the API. And run a periodic reconciliation sweep anyway, because some events will be missed and you will not know which.
pythonthe four rules, in about twenty lines
from fastapi import FastAPI, Request, HTTPException, BackgroundTasks
import hmac, hashlib, os

app = FastAPI()
SECRET = os.environ["WEBHOOK_SECRET"].encode()
seen = set()                      # in production: Redis with a TTL

@app.post("/webhooks/confluence")
async def receive(request: Request, background: BackgroundTasks):
    raw = await request.body()

    # 1. Verify -- compare_digest, not ==, to avoid a timing side channel
    expected = hmac.new(SECRET, raw, hashlib.sha256).hexdigest()
    if not hmac.compare_digest(expected, request.headers.get("x-hub-signature", "")):
        raise HTTPException(401, "bad signature")

    event = await request.json()

    # 2. Deduplicate -- retries WILL arrive for events you already handled
    if event["eventId"] in seen:
        return {"ok": True, "duplicate": True}
    seen.add(event["eventId"])

    # 3. Acknowledge NOW, work later. A slow 200 gets you retried or disabled.
    background.add_task(reindex_page, event["page"]["id"])
    return {"ok": True}

def reindex_page(page_id: str):
    # 4. The webhook is a HINT. Fetch current state -- the body may be stale,
    #    partial, or out of order relative to other events for the same page.
    page = confluence_api.get_page(page_id, expand="body.storage,version")
    upsert_chunks(page)

MCP — the model asks

MCP (Model Context Protocol) is the newest of the three and the most misunderstood. It is not a replacement for APIs. It is a standard way to describe tools to a language model, so the model can decide for itself when to use them.

The problem it solves, stated concretely

Before MCP: to let a model search Confluence, you hand-wrote a tool definition — name, description, parameter schema — plus the code to call the API and format the result. Then you wrote it again for Jira. And again for your database.

And then again in a different shape for every agent framework you used, because each had its own tool format.

MCP standardises that description. Write one MCP server for Confluence and every client that speaks the protocol can use it — Claude Code, a chat assistant, an agent framework, an IDE.

The analogy that fits: OpenAPI describes a REST API to a programmer. MCP describes a capability to a model. Underneath, the MCP server is calling the ordinary REST API — it is a translation layer, not a new transport.

An MCP server exposes three kinds of thing: tools (actions the model may take — search, create, update), resources (content it may read), and prompts (pre-written templates a user can invoke). Tools are what people mean in practice.

pythona complete MCP server — the docstring does most of the work
from mcp.server.fastmcp import FastMCP
import httpx, os

mcp = FastMCP("confluence")

@mcp.tool()
def search_confluence(query: str, space: str | None = None, limit: int = 5) -> str:
    """Search Confluence pages by text. Returns titles, URLs and excerpts.

    Use this when the user asks about internal documentation, policies or
    runbooks. Prefer a specific `space` when the user names a team.
    """
    # ^ THIS DOCSTRING IS THE PROMPT. It is what the model reads to decide
    #   whether to call the tool and with what arguments. Vague description,
    #   vague tool use -- this is the single highest-leverage thing you write.
    cql = f'text ~ "{query}"' + (f' AND space = "{space}"' if space else "")
    r = httpx.get(
        f"{os.environ['CONFLUENCE_URL']}/wiki/rest/api/search",
        params={"cql": cql, "limit": limit},
        headers={"Authorization": f"Bearer {os.environ['CONFLUENCE_TOKEN']}"},
    )
    r.raise_for_status()
    return "\n\n".join(
        f"{h['content']['title']}\n{os.environ['CONFLUENCE_URL']}{h['url']}\n{h.get('excerpt','')}"
        for h in r.json()["results"]
    )

if __name__ == "__main__":
    mcp.run()

The tool description is a prompt, and it is the part people underinvest in.

The model has never seen your system. All it has is the tool's name, its description, and its parameter names.

Write when to use it and when not to. ‘Search Confluence’ gets you a tool called at random. ‘Use this when the user asks about internal documentation, policies or runbooks’ gets you one called correctly.

If an agent is using your tools badly, the description is the first place to look and almost the last place people check.

Worked example: a chatbot over Confluence

This is the request that motivated the chapter, and it is a good one because the two architectures pull in opposite directions.

A CONFLUENCE CHATBOT — TWO ARCHITECTURES, AND MOST TEAMS NEED BOTH A — INDEX IT (the retrieval pipeline from chapter 12) PULLAPI + since-dateCHUNKper page sectionEMBED+ store ACLsSEARCHfiltered by userANSWERwith page links ✓ Fast (~1s)   ✓ Searchable across thousands of pages   ✓ Permission pre-filtering is possible ✗ As stale as your last sync   ✗ You own the index, the deletions and the re-crawl B — GIVE THE MODEL A TOOL (MCP) USER ASKSin the chatMODEL DECIDESto call search_confluenceMCP SERVERcalls the live APIANSWERfrom what came back ✓ Always current   ✓ No index to build, delete or re-crawl   ✓ The model can follow up and search again ✗ Slower (2–5s)   ✗ Only as good as Confluence’s own search   ✗ Permissions depend on WHOSE TOKEN it uses THE QUESTION THAT DECIDES B, AND IT IS NOT A TECHNICAL ONE: Does the MCP server call Confluence as the asking user, or as one service account that can read everything? A shared service account is the default in most tutorials, and it hands every user the permissions of the most privileged one.
The red box is the decision that matters, and it is an identity decision rather than an engineering one.

Architecture A — index it. The retrieval pipeline from chapter 12. Pull pages through the REST API on a schedule, chunk them by section, embed, store the space and page permissions alongside, and search with a permission pre-filter.

pythonConfluence sync — note what is captured alongside the text
# Incremental sync -- pull only what changed. The full crawl is for day one.
def sync_since(last_run_iso: str):
    start = 0
    while True:
        r = httpx.get(
            f"{BASE}/wiki/rest/api/content/search",
            params={
                "cql": f'type=page AND lastmodified > "{last_run_iso}"',
                "expand": "body.storage,version,space,restrictions.read.restrictions.group",
                "limit": 50, "start": start,
            },
            headers=AUTH,
        ).json()

        for page in r["results"]:
            upsert_chunks(
                page_id  = page["id"],
                title    = page["title"],
                url      = f"{BASE}/wiki{page['_links']['webui']}",
                text     = html_to_text(page["body"]["storage"]["value"]),
                updated  = page["version"]["when"],       # for the staleness filter
                acl      = read_groups(page),             # for the permission pre-filter
            )

        if len(r["results"]) < 50:
            break
        start += 50

# Add the webhook so edits land in seconds instead of waiting for the nightly run.
# Keep the scheduled sweep anyway -- webhooks get missed, and you will not know which.

Architecture B — give the model a tool. The MCP server above. No index, always current, and the model can search again when its first attempt misses — which is a genuine advantage a fixed retrieval step does not have.

The permission question, which decides whether B can go live at all

Whose credentials does the MCP server use?

A shared service account is what almost every tutorial does, and it means every user of the chatbot effectively has the permissions of that account. If it can read the HR space, so can everyone. Acceptable only if the corpus is genuinely open to all staff, and you should say that out loud rather than assume it.

Per-user credentials — the server calls Confluence with the asking user's own token — is correct, and it is the harder build: you need an OAuth flow, token storage and refresh per user.

This is the same problem as the pre-filter versus post-filter decision in chapter 13, wearing different clothes. It is also the single most common reason an internal chatbot pilot cannot be promoted to production.

Most real systems end up with both, and the split is sensible: index for search, because it is fast and permission-filterable across thousands of pages; a live tool for freshness, so the model can fetch the current version of a specific page it has already found. And a webhook to keep the index honest between syncs. All three, each doing what it is good at.

Choosing, in one table

NeedUse
Get something right nowAPI
React the moment something changesWebhook — plus a reconciliation sweep, because some will be missed
Let a model decide when to fetch somethingMCP
Search across thousands of documents fast, with permissionsIndex it — retrieval, not a live tool call
Guarantee the answer reflects the current versionA live tool call — an index is always as stale as its last sync

The sentence worth remembering: an API is you asking, a webhook is you being told, and MCP is the model asking. MCP does not replace the API — it wraps one so a model can use it without you hand-writing a tool definition for every client.

Check yourself

What is the one distinction between the three?Who starts the conversation. With an API you ask; with a webhook you are told; with MCP the model asks.
Why not just poll an API?Poll often and almost every request discovers nothing changed; poll rarely and you find out late. Webhooks exist to close that gap.
What are the four webhook rules?Verify the signature, respond immediately and work in the background, be idempotent on the event ID, and treat the payload as a hint — fetch current state and run a reconciliation sweep anyway.
What does MCP actually standardise?The description of a tool to a model. OpenAPI describes a REST API to a programmer; MCP describes a capability to a model. The server calls the ordinary API underneath.
What is the highest-leverage part of an MCP server?The tool description. It is a prompt — the model has only the name, description and parameter names. Say when to use it and when not to.
Index Confluence or call it live?Index for fast permission-filtered search across thousands of pages; call live for guaranteed freshness on a specific page. Most real systems do both, with a webhook keeping the index honest.
What decides whether a live MCP tool can go live?Whose credentials it uses. A shared service account gives every user the permissions of that account; per-user tokens are correct and are a harder build.
← Prompting, for people who have to shipBuilding an MCP server in an enterprise →