Document: simulation/MARBELL-SIMULATION-QUEUE.md, Vantasner AG device program, 2026-10-21.
Scope: design of the round-the-clock, agent-operated simulation queue that runs on Marbell bare-metal servers and feeds the 12-month device program. Companion toolchain document: simulation/TOOLCHAIN.md.
All throughput, runtime, and cost numbers beyond the fixed givens below are estimates and are marked as such. Software licenses were checked by web search on 2026-10-21.
1 Fixed givens
- Server rate: CHF 1500 per server per month on Marbell bare-metal.
- Slot count: 16 agent-plus-simulation slots per server.
- Derived slot rate: CHF 93.75 per slot-month (1500 divided by 16; arithmetic, not an estimate).
- Program horizon: 12 months to a working device. The queue exists to narrow the lab campaign's parameter space, never to run decorrelated science.
2 Assumptions (all estimates)
| Assumption | Value | Note |
|---|---|---|
| Marbell server shape | 64 physical cores, 512 GB random access memory, 8 TB non-volatile memory express (NVMe) storage, 10 GbE network (estimate) | The CHF 1500 covers amortized hardware, colocation, power, and bandwidth (estimate) |
| Agent reasoning model | Hosted application programming interface (API) call, one large language model (LLM) per slot | Token costs sit outside the CHF 1500 server rate and are budgeted separately in the program plan (assumption) |
| Agent runtime footprint | 1 reserved core and 16 to 32 GB per slot (estimate) | Agents are input/output bound: they mostly wait on LLM responses, so their cores free up for compute |
| Server compute pool | 48 of 64 cores shared by the 16 slots under weighted fair scheduling (estimate) | Per-slot baseline entitlement 3.0 core-equivalents; bursty effective use about 3.2 (estimate) |
| Month | 720 hours; duty factor 85 percent after restarts, queue overhead, maintenance (estimate) | About 2000 usable compute core-hours per slot-month (estimate) |
| Accelerators | No graphics processing units (GPUs) assumed; machine-learning potential training sized for central processing units (estimate) | A GPU option would raise class K12 throughput (estimate) |
3 Architecture
3.1 Slot anatomy: one AI agent per slot
Each of the 16 slots runs one autonomous agent: a Python runtime with scoped tool access to the queue API, the shared results database, the input-generation repositories, and a per-slot filesystem. The four functions of the loop are staffed by persona rotation, one persona per slot per week:
- Literature triage agent. Reads the prior-art corpus and monitored new literature; extracts parameter ranges (loading ratios, drive frequencies, particle sizes, screening potentials); files job requests into the intake queue with provenance citations.
- Job generation agent. Converts job requests into validated simulation inputs through the ASE (Atomic Simulation Environment) job factories, Elmer solver-input templates, and the Python kernel library; runs one-minute smoke checks locally; submits to the class lane.
- Run supervision agent. Watches the scheduler, restarts from checkpoints on failure, enforces wall-clock budgets, and escalates repeated failures to the fleet coordinator.
- Result ingestion agent. Parses outputs, runs acceptance checks, writes results rows and artifact checksums, and drafts the weekly parameter brief for the lab campaign.
3.2 Server software stack
| Component | Choice | Role | License (checked 2026-10-21) |
|---|---|---|---|
| Queue broker | Valkey | Priority lanes per job class, agent-to-agent bus | BSD-3-Clause |
| Scheduler | Dask distributed | One distributed scheduler per server over the 48-core pool; fleet-aware work stealing across servers | BSD-3-Clause |
| Results database | PostgreSQL | Shared, replicated job and results store with JavaScript Object Notation (JSON) columns | PostgreSQL License |
| Input generation | ASE job factories, Elmer templates, Python kernels | Reproducible inputs keyed by commit plus configuration hash | LGPL / GPL / BSD |
| Simulation codes | Quantum ESPRESSO, GPAW, CP2K, LAMMPS, PRISMS-PF, MOOSE, FEniCSx, Elmer, LIGGGHTS-PUBLIC, phonopy, MACE | See TOOLCHAIN.md | GPL / LGPL / BSD / Apache-2.0 |
| Execution isolation | Pinned containers executed with the Linux container runtime available on the host | Identical job environment on every server | n/a (distribution method) |
| Observability | Prometheus and Grafana | Fleet health, per-class throughput, cost meters | Apache-2.0 / AGPL-3.0 |
| Configuration | Ansible playbooks | Single-command server rebuild | GPL-3.0 |
3.3 Fleet topology
- Server 1 is the head: fleet coordinator agent, PostgreSQL primary, campaign planner, observability aggregation.
- Servers 2 to N are workers: PostgreSQL replicas, local Valkey for the server-level lanes, full toolchain containers.
- The coordinator re-plans weekly from the state of the results database, not from human slide decks.
3.4 Job lifecycle
- Request: a triage record with target observable, tool class, and citation provenance.
- Specification: a JSON record with class, code, resource request, convergence tolerances, and a unique idempotency hash built from the input-factory commit, the configuration hash, and the container digest. Duplicate hashes are dropped, which makes the whole fleet safely restartable.
- Queue: the job enters its class lane (K0 to K12, section 4).
- Run: containerized execution under Dask; checkpoints at fixed wall-clock intervals for long MD and DFT jobs.
- Acceptance checks: convergence flags, energy drift bounds, conservation checks, and physics anchors (lattice constants, bulk moduli, known rate integrals) evaluated automatically at ingest.
- Ingestion: results rows plus artifact checksums into PostgreSQL; the weekly parameter brief regenerates.
- Failure path: two automatic retries with checkpoint resume, then persona escalation (supervision agent to coordinator to human on-call).
3.5 Results database
Sketch of the shared store:
jobs: id, class, specification (JSON), provenance (JSON), state, timestamps.results: job id, observable, value, unit, uncertainty, context (JSON: composition, strain, drive, carrier).artifacts: path, checksum, size.briefs: week, parameter brief document, diff against previous week.
The database is the single source of truth for the monthly decision link to the lab campaign (section 7).
4 Job-type taxonomy with cost and runtime class
Cost per job = core-hours divided by 2000 usable core-hours per slot-month, times CHF 93.75. All values in this section are estimates.
| Class | Tool | Typical job | Cores | Wall-clock | Core-hours | CHF per job |
|---|---|---|---|---|---|---|
| K0 | Python kernels | Gamow/screening scan point, S-factor integral, spectrum kernel | 1 | 5 to 30 min | 0.1 | 0.005 |
| K1 | LAMMPS (EAM) | 10^6 atom contact-pulse transient | 16 | 6 h | 96 | 4.50 |
| K2 | LAMMPS (ReaxFF) | Ti-H/Zr-H interface or fracture cell | 32 | 24 h | 768 | 36.00 |
| K3 | Quantum ESPRESSO | 64-atom relaxation or strain point | 32 | 8 h | 256 | 12.00 |
| K4 | Quantum ESPRESSO + phonopy | Phonon + quasi-harmonic volume sweep | 32 | 48 h | 1536 | 72.00 |
| K5 | GPAW | Hybrid-functional interface slab | 48 | 72 h | 3456 | 162.00 |
| K6 | CP2K | 1000-atom ab initio MD trajectory (5 to 20 ps) | 48 | 96 h | 4608 | 216.00 |
| K7 | PRISMS-PF / MOOSE | 2D mechanics-coupled precipitation front | 16 | 24 h | 384 | 18.00 |
| K8 | PRISMS-PF / MOOSE | 3D precipitation under oscillatory strain | 48 | 48 h | 2304 | 108.00 |
| K9 | Elmer | Horn-plus-cell eigenfrequency or driven acoustics | 16 | 6 h | 96 | 4.50 |
| K10 | FEniCSx | Piezo grain time-domain elastodynamics + electrostatics | 8 | 4 h | 32 | 1.50 |
| K11 | LIGGGHTS-PUBLIC | DEM force-chain packing sweep | 32 | 12 h | 384 | 18.00 |
| K12 | MACE | Potential training run (CPU) | 48 | 24 h | 1152 | 54.00 |
5 Throughput model
Formula: jobs per slot-month per class = 2000 usable core-hours divided by core-hours per job, with K0 additionally capped at 3000 per slot-month by orchestration overhead (all estimates). Per-server figures multiply by 16 slots. Classes compete for the same pool; the campaign mix, not the ceiling, sets realized counts.
| Class | Jobs per slot-month | Jobs per server-month (16 slots) |
|---|---|---|
| K0 | 3000 (capped) | 48,000 |
| K1 | 20 | 333 |
| K2 | 2.6 | 41 |
| K3 | 7.8 | 125 |
| K4 | 1.3 | 21 |
| K5 | 0.6 | 9 |
| K6 | 0.4 | 7 |
| K7 | 5.2 | 83 |
| K8 | 0.9 | 14 |
| K9 | 21 | 333 |
| K10 | 62 | 1000 |
| K11 | 5.2 | 83 |
| K12 | 1.7 | 28 |
Illustrative mixed portfolio per server-month under a balanced campaign plan (estimates): 1000 K0, 60 K1, 15 K2, 80 K3, 12 K4, 4 K5, 4 K6, 40 K7, 6 K8, 120 K9, 250 K10, 40 K11, 4 K12. The mix is re-balanced weekly by the coordinator.
6 Campaign plan for 4, 8, and 16 servers
| Scale | Servers | Slots | Cost per month | 12-month cost | Character |
|---|---|---|---|---|---|
| A | 4 | 64 | CHF 6000 | CHF 72,000 | Foundations: DFT, kernels, MD, 2D phase field, horn acoustics; 3D phase field quarterly; one MLIP generation |
| B | 8 | 128 | CHF 12,000 | CHF 144,000 | Standard plan as scheduled below: weekly 3D phase field, full-cell FEM, four MLIP generations, full matrix sweeps |
| C | 16 | 256 | CHF 24,000 | CHF 288,000 | Everything in B plus replicate ensembles, 20 to 1000 kHz frequency sweeps, and live assimilation of lab data with weekly re-planning |
Monthly server cost equals server count times the fixed CHF 1500 rate. The month-by-month plan below is written on the Scale B basis; Scale A runs the same sequence with the reductions stated, Scale C with the additions stated. Job counts are estimates.
| Month | Campaign focus | Dominant classes | Est. jobs (Scale B) | Lab parameters narrowed |
|---|---|---|---|---|
| 1 | Queue commissioning; DFT foundations for PdD0.7, TiD2, ZrD2, NiHx; screening kernels v1; first horn model | K3, K4, K0, K9 | 800 K3, 120 K4, 5000 K0, 200 K9 | Hydride shortlist for stage 1; DFT anchor set; horn first resonance |
| 2 | Strain scans and phonons around the hydride transitions; potential benchmark phase; screening fit v1 | K4, K3, K1, K0 | 200 K4, 400 K3, 60 K1, 8000 K0 | Temperature window; thermal expansion mismatch; kernel benchmark against measured metallic screening potentials |
| 3 | Transient MD ensembles; DEM force chains v1; FEniCSx piezo grain kernel v1 | K1, K2, K11, K10 | 300 K1, 100 K2, 150 K11, 800 K10 | Drive amplitude ceiling; particle size window; contact-stress pulse distributions |
| 4 | Concept matrix scans: 4 hydrides x 4 piezoelectrics x carrier shortlist | K0, K9, K11, K3 | 20,000 K0, 400 K9, 200 K11, 300 K3 | Piezoelectric composition; volume fraction; carrier shortlist (dry bed versus deuterated oil versus electrolyte) |
| 5 | Carrier decision support: field-survival arithmetic, Debye screening, loaded acoustic models | K10, K9, K0 | 1500 K10, 300 K9, 10,000 K0 | Carrier choice; gap-field budget feasibility; coating and fit tolerances |
| 6 | MLIP dataset generation and first training; 2D phase-field fronts | K6, K12, K7 | 60 K6, 20 K12, 400 K7 | Interface mobilities; ReaxFF replacement for shock cells |
| 7 | 3D phase field under oscillatory strain; full-cell coupled FEM | K8, K9, K7 | 100 K8, 200 K9, 200 K7 | Loading ramp protocol; duty cycle; resonance tracking versus temperature |
| 8 | Drive frequency and duty-cycle sweeps | K9, K8, K1 | 500 K9, 80 K8, 200 K1 | Drive frequency final choice; power ceiling; duty cycle |
| 9 | Lab bring-up support: as-built cell models from measured geometry | K9, K10, K0 | 600 K9, 2000 K10, 5000 K0 | As-built resonance prediction; expected contact charge; thermal balance |
| 10 | Signal-class predictions: spectra kernels over the surviving matrix | K0, K2, K6 | 30,000 K0, 50 K2, 20 K6 | Detector exposure plan; proton line versus continuum expectations; minimum detectable yield targets |
| 11 | Design lock: sensitivity sweeps over the 12 design parameters; margin list | K8, K9, K3, K0 | 60 K8, 300 K9, 200 K3, 20,000 K0 | Final parameter brief version 1.0; margin list for the lab |
| 12 | Campaign support: assimilate first device data, correct contact-stress and field models | K10, K9, K0 | 1000 K10, 200 K9, 10,000 K0 | Live parameter corrections feeding the next build cycle |
Scale A adjustments (estimates): K8 runs quarterly instead of weekly (about 20 per month total), K12 runs once in month 6 and once in month 11, K5 counts drop by half, everything else holds. Scale C adjustments (estimates): K8 quadrupled, K11 tripled for replicate force-chain ensembles, K0 multiplied five-fold for full frequency and matrix sweeps, and months 9 to 12 add a nightly assimilation lane that ingests lab diagnostics and re-runs K10 plus K0 against measured data.
7 Decision link to the lab campaign
Each simulation campaign exists to narrow named lab parameters. The coordinator agent regenerates this table weekly from the results database; the version below is the month-1 plan.
| Simulation campaign | Lab parameter narrowed | Delivered | Decision owner |
|---|---|---|---|
| DFT relaxations and strain scans (K3) | Hydride choice per stage: deuteron density versus price versus mobility (PdD0.7 versus TiD2 versus ZrD2 versus NiH) | Months 1 to 2 | Lab lead |
| Phonons and quasi-harmonic expansion (K4) | Temperature window; thermal expansion mismatch tolerances for particle coating and cell fit | Month 2 | Lab lead |
| Screening and tunneling kernels (K0) | Expected signal class; minimum detectable yield; detector exposure plan | Months 2 and 10 | Diagnostics lead |
| Transient MD (K1, K2, K6) | Maximum drive amplitude; particle size window (1 to 10 micrometer baseline) | Month 3 | Lab lead |
| DEM force chains (K11) | Piezoelectric volume fraction (about 10 percent baseline); packing fraction; hydride-to-piezo size ratio | Month 4 | Formulation lead |
| FEniCSx piezo kernels (K10) | Carrier fluid choice (field survival); gap budget against the 10^8 to 10^9 V/m design target | Month 5 | Device lead |
| Elmer horn and cell acoustics (K9) | Horn geometry; drive frequency (20 versus 40 kHz class); power ceiling below cavitation onset | Months 1, 8, 9 | Device lead |
| Phase field (K7, K8) | Loading ramp protocol; duty cycle (drive-on fraction) | Months 7 to 8 | Lab lead |
| Assimilation lane (K10, K0) | Live corrections of contact-stress and field models from first device data | Month 12 onward | Device lead |
8 Operating notes
- Weekly re-planning: the coordinator agent publishes a report every Monday 06:00 from database state; humans steer by exception.
- Escalation: repeated job failure moves supervision agent to coordinator to human on-call within one hour.
- Every number that reaches the lab carries provenance (code version, container digest, input hash) and an estimate tag where it is an estimate.
- The queue publishes only results that passed the automatic acceptance checks at ingest; failures stay in the database with their failure class for the weekly report.
- Data hygiene: nightly snapshots of the filesystem, offsite backup of the database and briefs, and a documented single-command rebuild of any server from the Ansible playbooks.
9 Cost summary for the 12-month program
| Scale | Servers | Slots | Monthly cost | 12-month total | Slot-months purchased |
|---|---|---|---|---|---|
| A | 4 | 64 | CHF 6000 | CHF 72,000 | 768 |
| B | 8 | 128 | CHF 12,000 | CHF 144,000 | 1536 |
| C | 16 | 256 | CHF 24,000 | CHF 288,000 | 3072 |
Costs follow from the fixed CHF 1500 per server-month rate; the useful-work differences between scales are the throughput estimates of section 5. Excluded from these figures: LLM API token spend (assumption: budgeted separately in the program plan), laboratory diagnostics, and any storage beyond the servers' own disks (estimate note).