Software Engineering · Chapter 25 of 31
GitHub Copilot, honestly
What the modes actually are, what it costs now that billing has changed, and what the evidence says about whether it helps.
Copilot is the assistant most people meet first, usually because their employer already pays for it. This chapter covers what the parts are, and then the harder question of whether it works — where the evidence is more interesting than either side's marketing.
First, a name collision that causes real confusion
Microsoft has attached the word Copilot to several unrelated products.
GitHub Copilot — this chapter. An assistant inside your code editor, billed per developer by GitHub.
Microsoft 365 Copilot — a completely different product. An assistant inside Word, Excel, Outlook and Teams, grounded in your company's own documents and email. Different licence, different price, different team buying it.
Copilot Studio — a builder for making your own assistants, again separate.
They share a brand and nothing else. If you are asked in an interview whether your company ‘uses Copilot’, the useful reply is to ask which one.
The modes, which are genuinely different things
| Mode | What it is | Where it runs |
|---|---|---|
| Completions | The grey text that appears as you type. Accept with tab | Locally triggered, small context. Free and unmetered on paid plans |
| Chat | A conversation panel. It answers; it does not change your files | Your editor |
| Edits | Chat that proposes changes across several files, which you approve one at a time | Your editor |
| Agent mode | Chat given tools — it can read files, edit them, run terminal commands. You watch it work, turn by turn | Your editor, your session, your permission prompts |
| Coding agent | A different thing entirely. You assign it a GitHub issue. It works on its own and opens a pull request for you to review | On GitHub's servers, not your machine. You are not watching |
The last two get confused constantly because both say “agent”. The distinction that matters: agent mode is a pair programmer sitting with you, and you interrupt it when it goes wrong. The coding agent is a colleague you hand a ticket to, who comes back later with a pull request. They are meant to be used together — prototype something with the first, hand the follow-up chores to the second.
What it costs, and the bit that just changed
The billing model changed on 1 June 2026, and most guides you will find predate it.
Copilot used to count “premium requests”. It now meters tokens — the units of text going in and out — against a credit allowance, at each model's own rate. Completions stay free and unlimited; chat, agents and code review draw from the pool.
And the safety net was removed. Under the old system, running out dropped you to a cheaper model. GitHub's own announcement says that fallback is gone — you now stop until the month resets or someone approves an overage budget.
This is the second billing change in Copilot's life. Assume there will be a third.
| Plan | Price | What you get |
|---|---|---|
| Free | $0 | 2,000 completions and 50 chat requests a month. Enough to decide whether you like it |
| Pro | $10/mo | Unlimited completions, $15 of credits, the full model picker, coding agent |
| Pro+ | $39/mo | $70 of credits, the largest models first, audit logs |
| Business | $19/user/mo | Organisation policy controls, audit logs, IP indemnity, and your code contractually excluded from training |
| Enterprise | $39/user/mo | The above plus indexing of your own codebase for chat |
| Max | $100/mo | $200 of credits — aimed at people running agents all day |
On which models you can pick: do not memorise a list. Over the nine months to August 2026, a new frontier model reached Copilot roughly every two to five weeks. The newest ones land on the higher plans first. Open the picker and look; any written list, including one in this chapter, is wrong by the time you read it.
Does it actually help?
Here the evidence is genuinely more interesting than the argument, because it does not line up neatly on either side.
The randomised trial, and what it does and does not say
In July 2025, METR ran the only proper randomised controlled trial in this area: 16 experienced developers working on their own large open-source projects, 246 real issues they chose themselves, each issue randomly assigned to allow AI or not.
They were 19% slower with AI — and the range was +2% to +39%, so a real slowdown, not noise. They had predicted 24% faster, and afterwards believed they had been 20% faster.
What it does not say is that these tools are useless. It is 16 senior people on codebases they know intimately — the case where a model has least to add. And METR ran a larger follow-up in 2026 whose point estimates lean the other way.
But be careful with that follow-up, because it is widely misreported. METR itself says the newer numbers are affected by selection bias — developers increasingly refused to be randomised into the no-AI group — and describes its own data as “only very weak evidence”. Anyone telling you METR reversed its finding is overclaiming what METR claims.
The durable lesson is the one from the original: your sense of being faster is not evidence of being faster. If it matters, measure it.
On security, the number has not moved. Testing over a hundred models on set coding tasks, roughly 44% of solutions introduced a vulnerability in the 2026 study — against 45% a year earlier. One percentage point. And the things people assume would fix it do not: bigger models scored 53% against 51% for smaller ones, and models specialised for coding did no better than general ones. The only factor that moved the needle was reasoning capability, and only from 51% to 56%. The best individual model still failed about one task in three.
The finding that should change what you do
Telemetry across 10,000 developers found that heavy AI adopters completed 21% more tasks and merged 98% more pull requests. Unambiguously more output. But in the same data, time to review a pull request rose 91%, the average pull request grew 154% larger, bugs rose 9% — and organisation-level delivery metrics stayed flat.
Writing code got faster. Reading it did not. So the saving was absorbed by a longer review queue instead of arriving as faster delivery. That is a vendor's own telemetry rather than an independent study, so weigh it accordingly — but it is measured behaviour rather than a survey, and it matches what the security number would predict.
So what should you actually do
Spend the time you save on review, not on more output. The bottleneck moved; put people where it moved to.
Keep pull requests small even though it is now easier not to. The 154% figure is the mechanism by which the gain disappears.
Never merge generated code you have not read. One in three tasks from the best model still went wrong on security, and generated code is confident and plausible, which is exactly what makes a skim insufficient.
Ask it to explain, not just to write. The strongest reported use is understanding unfamiliar code, and that is the mode with no security downside at all.
And do not trust your own sense of speed. That is the one thing the randomised evidence is unambiguous about.
Check yourself