High-concurrency small jobs turn into an orchestration project
Running many small concurrent jobs on Spark becomes a scheduling problem solved with thread pools. A stateless engine absorbs the bursts without the workaround.
THE PROBLEM
Running many small concurrent ETL or agent jobs on Spark is treated as an orchestration problem to solve with thread pools and schedulers, because the engine is built for large jobs rather than high concurrency of small ones.
THE PROBLEM ANATOMY
Who it hits
Owns the Spark configuration and the throughput target.
Owns the notebooks and jobs, and writes thread-pool code to fan them out.
Owns the cluster the concurrent jobs contend on.
The situation
A team wants to run many small jobs at once, each doing little work. On Spark the advice is that this is an orchestration problem: wrap the work in a function and fan it out with a thread pool, or lean on the scheduler, rather than expecting the engine to handle the concurrency itself.
That turns a runtime concern into application code. Every team rebuilds the same fan-out plumbing, and the cluster still contends when the small jobs pile up.
The specific challenges
- Built for big jobs: the engine favors large tasks over many small ones
- Orchestration workaround: concurrency is handled in thread-pool code, not the engine
- Repeated plumbing: each team rebuilds the same fan-out logic
- Cluster contention: small jobs still compete for the same shared resources
WHAT IT COSTS
PRACTICAL
- Custom fan-out: engineers write and maintain thread-pool code per workload
- Tuning time: cluster sizing gets revisited every time concurrency grows
BUSINESS
- Slower delivery: work goes into plumbing instead of pipelines
- Capped throughput: small-job concurrency hits a wall the engine will not clear
EMOTIONAL
- Reinvented wheels: the same orchestration hack shows up in every project
- Quiet frustration: a runtime problem lands on application developers
THE WAY OUT
A lakehouse compute engine with stateless per-vCPU scaling
Swap in a pilot of e6data's query engine alongside the existing Spark stack to:
Absorb the bursts: stateless services add and remove single vCPUs as concurrent jobs arrive, so many small jobs run at once without a thread-pool wrapper. The jobs and open tables stay in place. What changes is that concurrency is handled by the engine, not the application.
Drop the plumbing: with the engine scaling on its own, the fan-out code and scheduler tricks stop being the way to reach throughput. The orchestration that still fits the work stays. What is removed is the per-workload concurrency hack.
Keep the stack: the tables, catalog, and existing Spark jobs stay as they are, and adoption is an endpoint change rather than a migration, needing only fast validation of the required integration mapping. What is added is headroom for small-job concurrency.
The result is small-job concurrency the engine handles on its own, instead of plumbing every team rebuilds. Setup and connection details are in the query engine documentation.
See how the Query engine absorbs bursts Share your problem, talk with a founding team member →Book a demo on your own workloads
Reach out to book a demo, share challenges you're facing, and tell us how this fits into what you're currently working on or thinking about.
Prefer to self-serve? Problems we're solving →