Infrastructure & Reliability · Middleware — File transfer · Chapter 30 of 33
Managed file transfer, from zero: why banks still move files
Why the batch file survived the API era, what FTP got wrong, what SFTP fixed, and what the word ‘managed’ adds — checkpoints, encryption at rest, audit, partners and schedules.
A bank that runs real-time payments, open-banking APIs and event streaming still moves an enormous share of its money in files. Salaries, supplier payments, pension payouts, direct debits, statements, reconciliations, regulatory returns: every day, thousands of files cross the bank’s boundary in both directions. The software that moves them safely is called managed file transfer, and it is the third family of middleware.
Why files survived
- Batch economics. A payroll of 4,000 employees is one file, one approval, one transmission and one set of controls — not 4,000 separate calls, each of which can fail on its own.
- The schemes define files. National payment schemes accept bulk payments as files in prescribed formats. In the US, the ACH network’s NACHA format uses fixed-width records of exactly 94 characters. In the UK, organisations submit Bacs files through the Bacstel-IP channel on a three-working-day cycle — input on day one, processing on day two, money moving on day three. India’s bulk credits and debits run through NACH, the National Automated Clearing House operated by the National Payments Corporation of India.
- Counterparties produce files. A corporate client’s finance system exports a payment file; it does not call APIs.
- Some work is naturally a file. A statement, a reconciliation, a regulatory return: each is a document with a beginning, an end and a total.
- The mainframe runs in batches. Overnight batch processing consumes and produces files, and moves them between data centres.
From FTP to managed file transfer
| Protocol | What it is | Its problem, or its strength |
|---|---|---|
| FTP | The File Transfer Protocol, from the early internet | Sends the password and the data in clear text; opens extra connections that firewalls struggle with. Never acceptable across a bank’s boundary |
| FTPS | FTP over TLS — the same protocol, encrypted | Encrypted, but still FTP’s awkward multi-connection design |
| SFTP | The SSH File Transfer Protocol: file transfer inside a Secure Shell (SSH) connection. Not FTP with an S added | One port, strong encryption, key-based authentication — the default for bank-to-corporate transfers |
| HTTPS | Upload and download over the web’s encrypted protocol | Easy for people through a browser portal; APIs for systems |
| AS2 | Applicability Statement 2: files sent over HTTP, signed and encrypted, with a signed receipt returned | The receipt — a message disposition notification — proves delivery, which is why supply-chain and invoicing traffic (Electronic Data Interchange, EDI) uses it |
| AS4 | A successor built on web-services standards | Used in, among other places, European electronic invoicing networks |
| Connect:Direct | IBM’s proprietary point-to-point transfer, from the mainframe world, now sold as IBM Sterling Connect:Direct | Checkpoint restart for very large files and native mainframe support; the bank data-centre workhorse |
The name trap is worth spelling out, because it causes real misconfigurations: SFTP and FTPS are different protocols with different ports, different authentication and different firewall rules. A partner who says “secure FTP” must be asked which one they mean.
What ‘managed’ adds
A protocol moves bytes. A bank needs rather more, and MFT products exist to supply it:
| Capability | What it means in practice |
|---|---|
| Guaranteed delivery and checkpoint restart | A 20 GB transfer interrupted at 18 GB resumes from the last checkpoint rather than starting again |
| Integrity | A checksum proves the file that arrived is the file that was sent |
| Encryption at rest | Files waiting on disk are encrypted, not only files on the wire |
| PGP | Pretty Good Privacy (today, the OpenPGP standard): the file itself is encrypted for the recipient and signed by the sender, so it stays protected on every server it passes through |
| Audit and non-repudiation | Who sent what, when, to whom, whether it was received — kept for the retention period regulators require |
| Scheduling and triggers | Send the statement file at 06:00; start a workflow the moment a file arrives |
| Workflows | Decrypt, verify, scan for malware, validate the format, rename, route — without custom scripts |
| Partner management | Onboarding hundreds of counterparties, each with their own protocol, keys, folders and schedule |
| Service-level monitoring | An alert when a file expected by 09:00 has not arrived — the failure no transfer log can show |
PGP in two sentences
The sender encrypts the file with the recipient’s public key, so only the recipient’s private key can open it, and signs it with the sender’s private key, so the recipient can prove who sent it and that it was not altered.
The point for middleware: a PGP-encrypted file stays unreadable on every server along the way — including a compromised MFT server — until the one system that holds the key decrypts it.
The anatomy of an MFT estate
Push, pull, and who opens the connection
Every file flow has a direction of data and a direction of connection, and they need not match. A bank can push a statement to a client’s server, or the client can pull it from the bank’s. The rule of thumb in banks is that the more trusted side opens the connection: the bank would rather dial out than accept connections in, and the gateway pattern above exists to make inbound flows behave as if they were outbound.
Four kinds of key to manage
| Key | Proves or protects | Who holds it |
|---|---|---|
| SSH host key | The identity of the SFTP server, so a partner knows it reached the bank | The bank |
| SSH user key | The identity of the partner connecting, instead of a password | Each partner, with the public half registered at the bank |
| PGP key pair | The file’s contents and its sender | Both sides, each for their own direction |
| TLS certificate | Identity and encryption for FTPS, HTTPS and AS2 | The bank and each partner |
Every one of these expires or needs rotating, and rotating a key shared with hundreds of partners is a coordination project, not a command — the same lesson as partner certificates in the web tier, multiplied.
Worked example: onboarding a corporate client onto host-to-host
Every MFT engineer runs this process dozens of times, and interviews for the role often ask for it step by step. The order matters, because each step depends on the one before.
| Step | What happens | What goes wrong if it is skipped |
|---|---|---|
| 1. Agree the flows | Which files, in which direction, in which format and version, at what volume, against which cut-offs | A file format nobody agreed, discovered on go-live day |
| 2. Decide who connects | Usually the client pushes payments to the bank over SFTP; statements are pushed back or pulled | Firewall rules opened in the wrong direction |
| 3. Open the network | The bank allow-lists the client’s source addresses on the gateway, and the client allow-lists the bank’s | An outage the day the client’s cloud provider changes its addresses |
| 4. Exchange keys | The client’s SSH public key is registered; the bank’s host-key fingerprint and both PGP public keys are exchanged, and every fingerprint is confirmed over a second channel, such as a call to a known number | A key substituted in transit — the classic interception attack, invisible afterwards |
| 5. Test connectivity | A dummy file in each direction | Discovering a network problem with real money in the file |
| 6. Certify the format | Test files validated against the bank’s schema; rejections and status reports rehearsed | The first rejected file arriving with nobody knowing how to read the status report |
| 7. Pilot | Low-value live payments, with approval thresholds set | A mistake at full volume |
| 8. Go live, monitored | Expected-file monitoring for every flow, and a named escalation contact on both sides | A missing file noticed by the client’s employees rather than by either team |
Step 4 is the one most often done carelessly, and the one whose failure is hardest to detect later. A public key that arrives by email and is installed without checking its fingerprint is a public key an attacker may have supplied.
Check yourself