Every morning at six, a financial-services client of ours dropped a CSV onto an SFTP landing zone. It came out of their core-banking system overnight and held the previous day's transactions — internal transfers, ATM movements, QR payments. Our job was unglamorous but unforgiving: read the file, enrich each transaction with the customer's name from a separate CIF lookup service, validate it, write it into an Oracle data warehouse, and leave an audit trail for every single record so that when something went wrong, ops could point at row 347 and know exactly what happened to it.
The first version was a For Each scope, and it was the right call. The file had about a thousand transactions. For Each iterated them one at a time, each iteration calling the enrich API and inserting one row. It ran in a couple of seconds, it was trivial to read, and when a record failed we caught it and logged it without taking down the rest. Nobody thought twice about it.
Then the bank turned on a new product line and the nightly volume crept from a thousand records to tens of thousands, and eventually a separate reporting feed asked us to push roughly a million transactions a night into Salesforce. The exact same pattern that had been boring and reliable started throwing java.lang.OutOfMemoryError: Java heap space, and the worker would die somewhere in the middle with no way to know how far it had gotten. That is the moment most teams discover that For Each and Batch Job are not interchangeable — they live at different scales, and the seam between them is where the production incidents happen.
What For Each actually does, and where it quietly breaks
For Each is a Mule scope that walks a collection and runs its child processors once per element. It is sequential by default — one element, then the next, on a single thread — and on each iteration it overwrites payload with the current element. That overwrite is the part people forget. If you need the original collection inside the loop, you reach for vars.rootMessage.payload, because payload is now just the one record you are looking at. There is also a one-based vars.counter that is genuinely useful for progress logging, the kind of tx 412/1000 line that makes a 6 a.m. support call survivable.
The strength of For Each is that each iteration is a full Mule scope. You can call HTTP, hit a database, drop a JMS message, branch on a Choice router, wrap a Try block — anything. That is exactly why it is the wrong tool for pure data shaping. If all you are doing is transforming records with no side effects, the DataWeave map operator is dramatically faster because it runs in-memory as a pure function with no per-iteration scope overhead. The rule I give people is blunt: if you are calling an external system per record, use For Each; if you are only reshaping data, use map and stop building orchestration around a transformation problem.
The trap is that For Each loads the entire collection into memory before it starts iterating. At a thousand records that is free. At a million it is an OOM waiting for a busy night. And because it runs in-memory with no persistence, a crash at record 600,000 means you have processed 600,000 records, have no checkpoint, and get to start over from zero. There is no resume. For a nightly job that already takes hours, that is not a bug you want to find in production.
batchSize is not Batch Job — get this straight first
Before going further, there is a naming collision that causes real damage, so it is worth being pedantic. For Each has a batchSize property. It has nothing to do with the Batch Job component. All batchSize does is hand your loop N elements per iteration instead of one.
This is genuinely useful and underused. On the warehouse insert, instead of iterating a thousand records and firing a thousand single-row inserts, I set batchSize="100" so each iteration receives an array of a hundred records and runs one bulk insert. Ten round trips to Oracle instead of a thousand. The same trick lines up neatly with downstream limits — Salesforce's Composite API tops out at 200 records per call, so batchSize="200" is the natural grouping when you are pushing through it. You still get the simplicity of For Each; you just stop paying the per-record network tax.
The takeaway is that For Each batchSize is "how many elements per iteration," the Batch Job is a separate long-running async processor, and the Batch Aggregator — which we will get to — is a sub-scope inside a Batch step. Three things, three meanings, one unfortunate shared word. I have reviewed code where a developer conflated them and produced logic that ran without error and was silently wrong, which is the worst kind of wrong.
When the dataset wins: Batch Job and its three phases
Once the data outgrows the heap, you move to Batch Job, which is built for exactly this. Instead of holding everything in memory, it streams records into a persistent queue on disk, dispatches them across worker threads in parallel, and — if configured for it — can resume after a crash instead of restarting. It runs as a lifecycle of three phases: Load and Dispatch reads from the source and pushes individual records onto the queue; Process runs the records through one or more Batch steps, where worker threads pull from the queue concurrently; and On Complete runs exactly once at the end with a result object carrying totalRecords, successfulRecords, and failedRecords.
For the million-record Salesforce feed, that maps cleanly. A streaming db:select with a sensible fetchSize pulls the day's transactions out of the source Oracle without loading them all at once. The first Batch step enriches each record against the CIF service. A second step uses an acceptExpression to filter down to only the high-value transactions the support team cares about — records that do not match are skipped, not failed. A third step wraps a Batch Aggregator at size="200" so the enriched records get pushed to Salesforce in bulk, two hundred Cases per Composite call, which turns a million API calls into roughly five thousand. A final step with acceptPolicy="ONLY_FAILURES" catches everything that failed upstream and writes it to a dead-letter table for retry. On Complete logs the tallies and emails a summary to the ops lead so there is a report waiting in the morning.
The Aggregator is the workhorse here. Inside a step it gathers N records into a single chunk so you can do one bulk operation instead of N small ones — bulk DB inserts, bulk API calls, batched file writes. Just size it against the real downstream limit and remember the memory cost rises with the chunk size per worker, and that depending on the driver, one bad record can fail the whole chunk rather than just itself.
The gotchas that cost real time
Two Batch Job behaviors burn people, and both come from assuming it behaves like ordinary flow code. The first is that On Complete is not a success callback — it runs always, including when every single record failed, because the job has "completed" in the lifecycle sense regardless of outcome. If you put "Sync SUCCESS" email logic in On Complete unconditionally, you will cheerfully report success on a night when nothing got through. The fix is one line of DataWeave: branch on payload.failedRecords and report honestly.
The second is that a failed record does not fail the job. By default a record that errors in a step is marked FAILED while the others keep going; the job only aborts when the failure count exceeds maxFailedRecords. That default of streaming past errors is usually what you want for an ETL load, but you have to set it deliberately — 0 to abort on the first failure, -1 to push through no matter how many fail, or a real threshold tied to your business rule. Contrast that with For Each, where an unhandled error throws and stops the entire flow unless you wrapped it in a Try. The two scopes have opposite failure instincts, and picking the wrong one means either a job that abandons good records or a job that quietly swallows a disaster.
There is also the subtler one: do not trust payload after a For Each. People assume the scope restores the original collection, and it does — but if your loop body ran a DB insert or an HTTP call, what you are looking at afterward may be the last response, not your data. Save the collection into a variable before the loop and read it back from there. Guessing about post-loop payload state is a debugging session you do not need.
The deciding principle, after all of it, is to size the tool to the data and not to the elegance of the diagram. Under a thousand records, For Each is the honest answer and a Batch Job is over-engineering that will run slower than the loop it replaced. Past ten thousand, or any time you need to survive a crash, run in parallel, or hand back a real success-and-failure report, Batch Job earns its complexity. The mistake is never which one you start with — it is forgetting to revisit the choice when the volume quietly changes underneath you.
Building or operating MuleSoft integrations? Our Salesforce team designs API-led architectures, builds Mule flows, and runs them in production. Get in touch ->
See our full platform services for the stack we cover.
