aeaether
$docs / runtime

Aether Scheduler - Quick Reference

Sends

A send is asynchronous whatever the number of actors: the payload is copied to the heap and handed to the scheduler, and a scheduler thread runs the handler while the sender goes on (#3112).

Cooperative Scheduler

On platforms without pthreads (WebAssembly, embedded) or when threading is explicitly disabled (-DAETHER_NO_THREADING), the cooperative scheduler (aether_scheduler_coop.c) provides the same API as the multi-core scheduler but runs everything on a single thread.

Key differences:

  • scheduler_init() forces num_cores = 1
  • scheduler_start() / scheduler_stop() are no-ops
  • A send queues (the mailbox, then the actor's inbox past 32 messages) and never runs a handler
  • scheduler_wait() polls all actors via aether_scheduler_poll() until no messages remain
  • All sends go to local mailbox (no cross-core routing)
  • Ask/reply polls until reply_ready

Selection: Controlled by PLATFORM=wasm|embedded or auto-detected from EXTRA_CFLAGS containing AETHER_NO_THREADING. Only one scheduler .c file is compiled.

# Test cooperative mode on native
make ci-coop

# Build for WASM via CLI
ae build --target wasm hello.ae

# Build for WASM via Makefile
make stdlib PLATFORM=wasm

Actor Timeouts

Actors with a receive ... after N clause have timeout_ns and last_activity_ns fields in ActorBase. The generated step function checks the clock via _aether_clock_ns() before mailbox_receive():

  1. If the mailbox is empty and timeout_ns > 0, last_activity_ns is set to the current time (start countdown)
  2. On the next poll, if (now - last_activity_ns) >= timeout_ns, the timeout handler fires and the timeout is disabled (one-shot)
  3. If a message arrives, both timeout_ns and last_activity_ns are reset to 0 (cancelled)

Both the multicore and cooperative schedulers poll actors with active timeouts even when their mailbox is empty.

Cooperative Preemption

Opt-in, zero cost when disabled. Two independent levels:

Scheduler-side (AETHER_PREEMPT=1): After each actor->step() call in the drain loop, the scheduler checks elapsed time via aether_now_ns(). If the batch exceeds the threshold (default 1ms, configurable via AETHER_PREEMPT_MS), it breaks out and moves to the next actor. Cost when disabled: zero (branch skipped).

Codegen-side (aetherc --preempt): Inserts sched_yield() at the top of every while and for loop body. A static int _aether_reductions counter (initialized to 10000) controls frequency, yields only when the counter hits zero. Note this counter is a single file-scope global, not thread-local, so in a multi-threaded actor program it is shared across all threads. Cost when --preempt is not passed: zero (code not generated).

Scheduler Usage

// Initialize with N cores (auto-detects hardware if 0)
scheduler_init(4);

// Register actors (assigned to preferred core)
scheduler_register_actor((ActorBase*)actor, preferred_core);

// Start processing
scheduler_start();

// Send messages (routed automatically)
for (int i = 0; i < 10000; i++) {
    Message msg = message_create_simple(1, 0, i);
    scheduler_send_remote(actor, msg, current_core_id);
}

// Wait for quiescence (non-destructive, can be called multiple times)
scheduler_wait();

// Can send more messages and wait again
for (int i = 0; i < 5000; i++) {
    Message msg = message_create_simple(1, 0, i);
    scheduler_send_remote(actor, msg, current_core_id);
}
scheduler_wait();

// At program exit: wait for quiescence, stop and join scheduler threads
scheduler_shutdown();

Actor Allocation

// NUMA-aware, 64-byte-aligned allocation with the derived-struct size.
// NULL means the allocation failed.
ActorBase* actor = scheduler_spawn_actor(preferred_core, step_fn, sizeof(MyActor));
if (!actor) return NULL;

// Release it with scheduler_release_actor, never free(). It takes the actor
// out of its core's table at once and reclaims it once no scheduler thread,
// table reader or per-actor thread can still hold it, so it is safe from the
// actor's own step, and never blocks: an actor with its own thread
// (auto_process) is only marked, and that thread ends it when it leaves its
// loop. Nothing may send to the
// actor after this: the memory stays allocated and marked released until a
// spawn of the same size reuses it, so a late send is dropped and counted
// (scheduler_released_sends), and a send after the reuse reaches the new
// actor.
scheduler_release_actor(actor);

Message Flow

sender -> scheduler_send_remote -> lock-free queue -> batch dequeue
       -> coalesce buffer -> redirect/deliver -> mailbox -> actor step

Same-core fast path:

sender -> scheduler_send_local -> mailbox (direct) -> actor step

Configuration

Message Coalescing

#define SCHEDULER_DRAIN_BATCH 512  // In multicore_scheduler.h

Adaptive Batch Size

#define MIN_BATCH_SIZE 64    // In aether_adaptive_batch.h
#define MAX_BATCH_SIZE 1024

Mailbox Size

#define MAILBOX_SIZE 32   // In actor_state_machine.h (must be power of 2)

Message Pool

#define MSG_PAYLOAD_POOL_SIZE 256  // In aether_send_message.c
#define MSG_PAYLOAD_MAX_SIZE 256   // Max pooled message size

Cross-Core Queue

#define QUEUE_SIZE 1024  // In lockfree_queue.h

Idle Detection

scheduler_wait() blocks until all messages have been processed (quiescence). It is non-destructive: scheduler threads continue running after it returns, so it can be called multiple times in a program (e.g., between test phases or message bursts). It uses per-core counters to detect idle state without atomic contention on the message-passing hot path:

scheduler_shutdown() calls scheduler_wait() internally, then stops scheduler threads and joins them. Call it once at program exit.

// Per-core counters in Scheduler struct
uint64_t messages_sent;      // Incremented on send (no atomic)
uint64_t messages_processed; // Incremented on process (no atomic)

// scheduler_wait() sums across cores
while (total_sent != total_processed) {
    // spin-wait
}

Messages sent from the main thread (before scheduler threads start) use a separate atomic counter. This path is infrequent.

Core Assignment and Migration

Actors are assigned to cores at spawn time using locality-aware placement: the runtime places each actor on the caller's core so that parent-child messaging stays local. Actors spawned from the main thread default to core 0, keeping all top-level actors co-located for efficient intra-group messaging. Actors spawned from within actor handlers (on scheduler threads) inherit the parent's core.

During operation, cross-core senders set a migrate_to hint on the target actor. The owning scheduler checks migration hints after processing messages in the coalesce buffer and migrates the actor to the sender's core. This co-locates communicating actors regardless of their initial placement.

Both migration and work stealing use ascending core-id lock ordering to prevent deadlock.

Best Practices

Do:

  • Let the scheduler handle core assignment and migration
  • Use scheduler_send_remote for all cross-core messages
  • Initialize migrate_to = -1 when creating actors manually
  • Ensure test actor structs match ActorBase field layout exactly

Do not:

  • Access an actor's mailbox from a thread other than its owning scheduler thread
  • Call mailbox_send directly on actors owned by other cores
  • Omit migrate_to from custom actor structs (causes struct layout mismatch)

Troubleshooting

Low Throughput

  • Verify compiler flags include -O3 -march=native -flto
  • Check that message pool is active (pool stats via aether_message_pool_stats)
  • Ensure batch size is appropriate for workload
  • Verify core count matches available hardware

High Latency

  • Tune SCHEDULER_DRAIN_BATCH for latency-sensitive workloads (lower = less batching delay)
  • Check progressive backoff thresholds
  • Avoid blocking operations in actor step functions

Crashes

  • Verify every actor struct starts with ActorBase's fields: expand AETHER_ACTOR_BASE_FIELDS first (or put an ActorBase member first) rather than copying the list, which drifts as fields are added. In order, they are active, id, mailbox, step, thread, auto_process, assigned_core, migrate_to, spsc_queue, reply_slot, step_lock, timeout_ns, last_activity_ns, dead, alloc_size, destroy_state, scheduler_owned, inbox, thread_parked, inflight.
  • Initialize migrate_to = -1 after actor creation
  • Use scheduler_send_remote instead of direct mailbox writes for cross-core messages
  • Call scheduler_shutdown() before process exit (waits for quiescence, stops and joins threads)

Files Reference

Headers

  • runtime/actors/actor_state_machine.h - Message struct, mailbox
  • runtime/scheduler/multicore_scheduler.h - Scheduler API, ActorBase, spinlock
  • runtime/scheduler/lockfree_queue.h - Cross-core message queue
  • runtime/actors/aether_spsc_queue.h - Same-core SPSC queue
  • runtime/actors/aether_adaptive_batch.h - Adaptive batch sizing

Implementation

  • runtime/scheduler/multicore_scheduler.c - Multi-core scheduler: coalescing, migration, work stealing
  • runtime/scheduler/aether_scheduler_coop.c - Cooperative single-threaded scheduler (WASM, embedded)
  • runtime/config/aether_optimization_config.h - Platform detection (Tier 0 AETHER_HAS_* flags)
  • runtime/actors/aether_send_message.c - Message sending, thread-local pools
  • runtime/actors/aether_actor_thread.c - Actor-owned thread loop
  • runtime/utils/aether_thread.h - Portable thread primitives (POSIX/Win32/no-op stubs)
  • runtime/aether_numa.c - NUMA topology detection and allocation

Tests

  • tests/runtime/test_scheduler.c - Scheduler correctness tests
  • tests/runtime/test_scheduler_stress.c - Concurrency stress tests
  • tests/runtime/test_scheduler_optimizations.c - Optimization validation