Embedding Aether as a Shared Library (--emit=lib)

aetherc --emit=lib (or ae build --emit=lib) compiles a .ae file into a dynamic library (.so on Linux, .dylib on macOS) with ABI-stable entry points that any FFI-capable language can call: C, Java (Panama), Python (ctypes or SWIG), Go (cgo), Ruby, Rust (extern "C"), and so on.

This document covers the v1 contract. For the broader design rationale, see aether-embedded-in-host-applications.md.

Quick example

// config.ae
import std.map

build_config(env: string, port: int) {
    m = map_new()
    map_put(m, "env", env)
    map_put(m, "port", port)
    return m
}

Build:

ae build --emit=lib config.ae
# produces libconfig.so (or libconfig.dylib on macOS)

Consume from C:

#include "aether_config.h"
#include <dlfcn.h>

typedef AetherValue* (*build_fn)(const char*, int32_t);

int main(void) {
    void* h = dlopen("./libconfig.so", RTLD_NOW);
    build_fn build = dlsym(h, "aether_build_config");
    AetherValue* cfg = build("prod", 8080);

    const char* env  = aether_config_get_string(cfg, "env");
    int32_t     port = aether_config_get_int(cfg, "port", 0);
    printf("%s on %d\n", env, port);   // prod on 8080

    aether_config_free(cfg);
    dlclose(h);
}

The flag matrix

--emit=csrc: distribute source, not N native libs (#996)

A single native lib can't be universal across OSes — ELF vs Mach-O vs PE/COFF are different loader formats. But the pre-native artifact can be: the generated C is portable, so --emit=csrc moves the "which platform" decision from distribution time (ship 6 natives) to the consumer's build:

ae build --emit=csrc mylib.ae -o mylib
# writes mylib.c (portable C) + mylib.h (aether_<name> prototypes)
#      + mylib.catalog.json (the describable ABI surface)

# compile it wherever, against the runtime:
cc -fPIC -shared mylib.c $(ae cflags) -o libmylib.so   # this host
# or feed mylib.c to WASM, or static-link it into a compiled host, or
# publish {mylib.c, mylib.h, mylib.catalog.json} as a content-addressable
# source package.

The .h is a normal C header (include guard + extern "C"); a consumer #includes it and calls aether_add(...) / aether_greet(...) directly. This is the enabling primitive for compile-on-install bindings (Python C-extension / Ruby-gem style) and a "source registry" story where every binding compiles the same hash-pinned source in its own toolchain. Follow-ups noted in #996: single-file amalgamation and expressing the required runtime .c set for a fully standalone external compile.

The JSON catalog (<name>.catalog.json)

The same aether_lib_meta() symbol catalog the .c carries in .rodata (see the reflection section below) is also written as a JSON document, so a binding generator in any language can consume it without dlopening a native artifact. It is a faithful serialization of the C struct, driven by the identical codegen tables, so the two can never drift; it is deterministic (source order, stable version strings) and human-diffable, which makes the source artifact content-addressable. Fields:

{
  "schema_version": "1.0",
  "aether_version": "0.0.0-dev",
  "primary_source": "mylib.ae",
  "capabilities": [],
  "functions": [
    { "aether_name": "add", "c_symbol": "aether_add", "signature": "(int, int) -> int", "source_file": "mylib.ae", "source_line": 4 }
  ],
  "closures": [],
  "constants": []
}

Naming

Every exported Aether function becomes aether_<name> in the library:

The internal sum, greet, etc. symbols are also present in the library with external linkage. Callers should use the aether_<name> entry points for ABI stability, the un-prefixed names are an implementation detail and may be hidden in a future version. See Symbol visibility matrix below for the full picture across emit modes.

Type mapping

Parameter and return types on exported functions are mapped to a fixed public ABI:

Functions whose parameters or returns use types outside this table (tuples, structs, closures, actor refs) compile but don't get an aether_<name> alias. aetherc prints a warning naming the skipped function. You can still use those types internally; they just can't cross the FFI boundary in v1.

Two private-helper conventions also opt out of the aether_<name> surface:

• Trailing-underscore: a function whose name ends in _ (e.g. record_start_, helper_, parse_line_) is treated as file-local. It's emitted with C static storage and gets no aether_* alias. This lets two .ae files in the same project declare their own record_start_ without colliding at link time and without leaking either into the public ABI. The convention is enforced by codegen, so adding the _ is enough, no annotation.
• Tuple-returning helpers: helper(n: int) -> { return n, n+1 } returns a _tuple_int_int struct that doesn't fit the public ABI table. The function itself is fine; the alias is skipped. Wrap with a single-value-returning function if it needs to be exposed across the library boundary.

The AetherValue* accessor API

Composite Aether values, maps, lists, generic ptrs, come back to the host as AetherValue* handles. The host walks them with the functions declared in runtime/aether_config.h:

// Map accessors
const char*  aether_config_get_string(AetherValue* root, const char* key);
int32_t      aether_config_get_int   (AetherValue* root, const char* key, int32_t default_value);
int64_t      aether_config_get_int64 (AetherValue* root, const char* key, int64_t default_value);
float        aether_config_get_float (AetherValue* root, const char* key, float  default_value);
int32_t      aether_config_get_bool  (AetherValue* root, const char* key, int32_t default_value);
AetherValue* aether_config_get_map   (AetherValue* root, const char* key);
AetherValue* aether_config_get_list  (AetherValue* root, const char* key);
int32_t      aether_config_has       (AetherValue* root, const char* key);

// List accessors
int32_t      aether_config_list_size       (AetherValue* list);
AetherValue* aether_config_list_get        (AetherValue* list, int32_t index);
const char*  aether_config_list_get_string (AetherValue* list, int32_t index);
int32_t      aether_config_list_get_int    (AetherValue* list, int32_t index, int32_t default_value);
int64_t      aether_config_list_get_int64  (AetherValue* list, int32_t index, int64_t default_value);
float        aether_config_list_get_float  (AetherValue* list, int32_t index, float   default_value);
int32_t      aether_config_list_get_bool   (AetherValue* list, int32_t index, int32_t default_value);

// Lifetime
void aether_config_free(AetherValue* root);

Ownership

Behaviour of missing keys / out-of-range indices

• Typed getters return the default_value the caller supplied.
• _get_string / _get_map / _get_list / _list_get_string return NULL.
• _list_size(NULL) returns 0; _has(NULL, k) returns 0.

Type contract

Aether maps and lists are untyped internally, values are stored as opaque void* with no runtime tag. The accessors reinterpret the stored value as the requested type; they do not verify it. That makes the script-to-host interface a straight FFI contract: document what the script writes at each key, and the host must read it back with the matching accessor. If the script stored an int at "port" and the host asks for a string, the host gets garbage, treat this exactly the way you would treat any other cross-language value exchange.

Capability-empty default

--emit=lib rejects .ae files that import:

• std.net, std.http, std.tcp networking
• std.fs filesystem
• std.os process / environment / shell

The rationale: a library embedded in a host process shouldn't have ambient network, filesystem, or process-spawning access. Those are capabilities the host grants; the script should only compute and return data. The host mediates I/O through whatever its own runtime provides.

std.map, std.list, std.string, std.json, std.math, and the other capability-free standard modules are allowed.

Opting in: --with=<capabilities>

Projects that are the host, code that compiles .ae and handwritten C into one binary, rather than embedding Aether as an untrusted user script, opt into specific capability categories:

ae build --emit=lib --with=fs           file.ae   # std.fs
ae build --emit=lib --with=net          file.ae   # std.net, std.http, std.tcp
ae build --emit=lib --with=os           file.ae   # std.os
ae build --emit=lib --with=fs,os        file.ae   # multiple, comma-separated
ae build --emit=lib --with=first-party  file.ae   # alias for fs,net,os
ae build --emit=lib --with=all          file.ae   # alias for fs,net,os

The gate stays default-deny: a build without --with=fs still rejects import std.fs. Unknown capability names are a hard error (typos shouldn't silently leave a gate closed). The categories mirror the banned-import groupings above, three buckets, chosen coarsely enough that opting in is an auditable event in a project's build invocation.

--with=first-party and --with=all both expand to fs,net,os. The two names are equivalent; pick whichever expresses intent better in your project. first-party reads as "this Aether code is trusted-as-first-party, give it everything"; all reads as a literal shorthand for the full set. Using either one is appropriate for tools like build systems and SDK generators where the .ae files ship with the toolchain itself; it is not appropriate for plugin / user- script scenarios, see "When NOT to use this" below.

When to use this. Systems code where the .ae files are first-party and version-controlled the same way as the .c files. The Aether-side file_open_raw is no different from the C side calling fopen there's no privilege boundary to police.

When NOT to use this. Anywhere the library could run untrusted Aether (user scripts, a plugin loader, a DSL evaluator). Leave the default in place; the host mediates I/O on the script's behalf.

Symbol visibility matrix

What gets static storage class versus external linkage in the generated .c file, by function origin × emit mode. This determines which symbols collide when multiple .c files compiled from different .ae sources are linked into one binary.

Two consequences worth highlighting:

Imported Aether wrappers are private to each TU. When module A and module B both import std.string, both generated .c files contain static const char* string_copy(...) a private copy each. Linking the two .o files together produces no duplicate-symbol errors, even on linkers that don't support -Wl,--allow-multiple-definition (notably macOS ld64). This is deliberate; see compiler/aether_module.c where cloned imported functions get clone->is_imported = 1, and compiler/codegen/codegen_func.c where the static prefix is emitted for functions guarded by func->is_imported || trailing_underscore_private.

Locally-defined functions retain external linkage. If two .ae files both define a function with the same name (e.g. both define helper), they DO collide at link time. --emit=lib adds an aether_<name> alias on top, but the un-aliased local symbol is still external. This is the symmetric opposite of the imported case and is intentional, local functions are the unit of inter-TU linking; the user program structures around their names.

Implication for downstream tools. Multi-TU binaries that import the same SDK module across many .ae files do not need -Wl,--allow-multiple-definition. The static-marking on imported wrappers handles deduplication at the source level. Tools building this shape can drop the link flag and gain macOS compatibility for free, ld64 (Apple's linker) silently rejects --allow-multiple-definition, and the static-marking removes the need for it.

Using SWIG to generate Java/Python/Ruby/Go bindings

The repo ships a minimal SWIG interface at runtime/aether_config.i. It wraps aether_config.h and marks aether_config_free as the destructor so target-language GC integrates cleanly.

# Python
swig -python -o aether_config_wrap.c runtime/aether_config.i
gcc -fPIC -shared $(python3-config --includes) aether_config_wrap.c \
    libconfig.so $(python3-config --ldflags) -o _aether_config.so

# Java
swig -java -package com.example.aether runtime/aether_config.i

# Ruby, Go, C#, ... same pattern with -ruby, -go, -csharp.

The generated bindings wrap AetherValue* as a proxy class per target language, Python gets AetherValue, Java gets a class AetherValue, etc. Users see idiomatic API calls; the opaque pointer is hidden.

See tests/integration/emit_lib_swig/ for a worked Python round-trip.

Per-call resource caps (#343)

Hosts loading untrusted Aether scripts bound the script's resource use via two TLS-backed knobs in include/libaether.h:

#include <libaether.h>

aether_set_memory_cap(64 * 1024 * 1024);   // 64 MiB ceiling
aether_set_call_deadline(50);              // 50 ms wall-clock

call_into_aether();                        // returns inside both bounds

if (aether_deadline_tripped()) {
    /* The deadline fired mid-call. The Aether call returned
     * early via the codegen tripwire. Any partial state is
     * the host's to interpret. */
}

Memory cap. Process-wide. The cap-aware allocators (std/string, std/collections, runtime/memory/{arena,pool}) check + account against a single _Atomic uint64_t counter. The counter tracks current usage, not high-water-mark, long-running guests that allocate-and-free in a loop don't trip the cap on cumulative churn. Allocations past the cap return NULL through the existing error-string convention; stdlib functions surface "out of memory: cap exceeded" up the call chain rather than aborting. Pass 0 to disable.

Wall-clock deadline. Per-thread monotonic (CLOCK_MONOTONIC). The codegen emits if (aether_caps_deadline_tripped()) { __aether_abort_call(); break; } at every for / while loop head under --emit=lib not under --emit=exe (zero overhead on non-sandboxed builds). On trip the sticky flag flips; subsequent loop heads in the same call exit too, so any depth of nested loops unwinds in O(N) tripwire-bounded breaks. Pass 0 to disable; pass a fresh ms value to clear the sticky flag and re-arm.

The trip is observable via aether_deadline_tripped() the host checks after the call to know whether the result was complete or truncated. Retrying a tripped deadline is the host's policy: re-arm + call again with a more generous budget, or surface Deadline to the user.

What's out of scope for v1

1. Callbacks held live, a host can't pass a closure into Aether and have it retained. If you want this (the ARexx / rules-engine model), track the "Shape B" design note in aether-embedded-in-host-applications.md.
2. Per-function capability grants, --with= flags are coarse (fs / net / os). Fine-grained gates like "allow file_open but not dir_delete" don't match any concrete threat model for the default-deny shape, and every additional flag is API surface.
3. Typed returns beyond void*, functions returning map/list come back as AetherValue* with no schema. Host knows the shape.

Kind discrimination + deep-free (aether_value_kind, aether_config_free_deep)

Aether's collections are intentionally untyped at the storage level, a map's value slot is void*, and the host knows the schema. That historically meant schema-mismatch bugs (script stores an int where a map was expected) degraded into host-side segfaults. v1 closes that gap with a magic-tagged kind discriminator.

Every HashMap and ArrayList carries a uint32_t _kind_magic as its leading field, set at construction and cleared on free. The runtime exposes:

typedef enum {
    AETHER_KIND_UNKNOWN = 0,
    AETHER_KIND_MAP     = 1,
    AETHER_KIND_LIST    = 2
} AetherKind;

AetherKind aether_value_kind(AetherValue* v);
int32_t    aether_value_is_map(AetherValue* v);
int32_t    aether_value_is_list(AetherValue* v);

These are safe to call on any pointer the host holds, including scalars that were intptr-cast to AetherValue* (e.g. (void*)(intptr_t)42 from a schema-mismatched _get_map call). The probe applies two layers of safety:

1. Low-address guard, the zero page + first 64 KiB are unmapped on every supported OS (Linux's vm.mmap_min_addr default is 65536; macOS reserves more; Windows likewise). Any pointer below 0x10000 is filtered before any deref attempt, so common intptr-cast scalars never reach the magic check.
2. 32-bit magic check, for higher-address values that survive the guard, the leading 4 bytes either match one of the two known constants (AETHER_KIND_LIST_MAGIC / AETHER_KIND_MAP_MAGIC) or the predicate reports UNKNOWN. False-match probability for a non-container value is 2 / 2³² ≈ 5 × 10⁻¹⁰.

aether_config_free_deep(root) builds on the predicate to walk a nested tree of containers and free each one in post-order. Scalars encountered along the way are left untouched (the host owns those separately). aether_config_free (shallow) auto-detects whether the root is a map or a list via the same predicate, so it now works on either container shape uniformly.

AetherValue* root = aether_build_config("prod", 8080);

if (aether_value_is_map(root)) {
    /* Defensive, confirm what the script actually returned. */
}

AetherValue* db = aether_config_get_map(root, "db");
if (aether_value_kind(db) == AETHER_KIND_MAP) {
    const char* host = aether_config_get_string(db, "host");
    /* … */
}

aether_config_free_deep(root);   /* nested map + list freed too */

Defense-in-depth: list_free and map_free clear the magic before releasing the struct's memory, so a use-after-free probe via aether_value_kind reads the cleared magic and reports UNKNOWN rather than a stale match on the freed-but-still-readable original value.

Reflection: the symbol catalog (aether_lib_meta), #403

Every --emit=lib artifact exports a single reflection entry point:

const AetherLibMeta* aether_lib_meta(void);

declared in runtime/aether_lib_meta.h. The returned struct describes every exported function, Aether name, C symbol, public signature, source file, source line, in a layout-stable, allocation-free form that any FFI consumer (Python ctypes, Java Panama, Ruby Fiddle, Node-API, hand-rolled dlsym) can walk directly.

typedef struct {
    const char* aether_name;     /* "double_int", "std.fs.copy"      */
    const char* c_symbol;         /* unmangled symbol to dlsym         */
    const char* signature;        /* "(int) -> int"                    */
    const char* source_file;      /* the .ae that defined it           */
    int         source_line;      /* 1-based                            */
} AetherLibFunction;

typedef struct {
    const char* schema_version;   /* "1.0" funcs; "1.1" closures; "1.2" consts */
    const char* aether_version;   /* compiler version                  */
    const char* primary_source;   /* the .ae passed to aetherc         */
    int                       function_count;
    const AetherLibFunction*  functions;
    int                       closure_count;   /* v2: closure-surface records */
    const AetherLibClosure*   closures;        /* NULL when closure_count==0 */
    int                       constant_count;  /* v3: exported-constant records */
    const AetherLibConstant*  constants;       /* NULL when constant_count==0 */
} AetherLibMeta;

No JSON, no parsing, no dynamic allocation, the struct is static const in the artifact and the catalog literally lives in .rodata. Schema is versioned; within 1.x only additive fields appear at the struct tail, so a "1.0" reader walks a "1.1" or "1.2" artifact unchanged (it reads function_count/functions and ignores the closure and constant slots; those fields were always present, just NULL before). schema_version is "1.0" for function-only artifacts, "1.1" once closure records are present, and "1.2" once exported-constant records (constant_count/constants, of type AetherLibConstant) are present. The function table includes every Aether-defined function the linker exports; functions skipped from the aether_<name> alias surface (unsupported types, tuple returns, trailing-underscore privates) are also omitted from it.

Closure-context records (v2, schema "1.1")

The flattened function table above is the foreign-FFI surface; it drops closures, real types, and the builder-DSL shape. The closures array is the complement: the closure surface a downstream Aether consumer needs to reconstruct a closure-with-context builder DSL at full fidelity, rather than the lowest-common-denominator C ABI a foreign host links against. Each record:

typedef struct { const char* name; const char* type; } AetherLibCapture;

typedef struct {
    const char* name;              /* param / closure name, or ""       */
    const char* role;              /* "builder"|"param"|"trailing-block"|"literal" */
    const char* enclosing_export;  /* the export it is reachable from   */
    const char* signature;         /* "|int, string| -> bool"           */
    int                     capture_count;
    const AetherLibCapture* captures;   /* NULL when capture_count == 0  */
    const char* source_file;
    int         source_line;
} AetherLibClosure;

Three sources populate it, all reachable from the set of top-level, non-imported, non-_-suffixed functions (which is broader than the ABI function table, a builder or a fn-parameter function is exactly what the ABI gate excludes, yet its bare symbol is still linkable by an Aether consumer):

• builder, an export that takes an injected _ctx (the builder keyword or the _ctx: ptr first-param convention). The "call me with a trailing block" signal; capture_count is 0.
• param, a closure-typed (fn) parameter of an export. name is the parameter, signature its |...| -> R shape.
• trailing-block, a trailing-block closure literal in an export's body.
• literal, any other hoisted closure literal in an export's body. For both of these, captures lists the enclosing variables the closure closes over with their rendered Aether types.

Closure-literal return types are best-effort: rendered when resolved, ? when the body type-checks lazily and the type isn't known at emit time. Captures and parameter types are exact.

Consuming a published library from Aether (import)

The records above are not just for inspection, they are what lets an Aether program consume a precompiled --emit=lib artifact as a first-class import, with the call site reading as if the library were compiled in the same cycle:

import std.map        // builder default factory (map_new) lives here
import gizmo          // resolves libgizmo.so, no gizmo.ae source needed

main() {
    println(gizmo.greet("world"))     // a function export
    s = gizmo.section("intro") { }    // a builder (trailing-block DSL) export
    println(s)
}

When ae run / ae build sees an import foo that has no foo source module but a libfoo.so / foo.so on the search path, it reads that artifact's aether_lib_meta() catalog and synthesizes a small Aether interface stub: an @extern("<c_symbol>") name(...) -> R for each function export, and a trailing-block builder wrapper for each builder record (forwarding the block's config map to the library function's (..., _builder) entry point). The stub is dropped onto the module search path, so the existing source-import machinery (typecheck, namespace prefixing, builder registration) rehydrates the library, no special call-site syntax. The artifact is linked in (absolute path + -rpath), and the host's -rdynamic + static libaether satisfy the .so's runtime symbols.

Scope of the current implementation:

• Function exports and builder DSL entry points are callable. A builder consumer also import std.map (the default map_new config factory is a std.map facility used at the call site).
• Higher-order exports (a function taking a closure fn parameter) are described in the metadata but not yet callable across the binary boundary, passing an Aether closure into an imported function is a follow-on (the closure ABI across the boundary).
• POSIX only for now (the prepass is gated on dlopen); Windows DLL hosting is a follow-up, matching ae lib-info.

ae lib-info <path> inspect any artifact

The ae CLI ships a turnkey reader that dlopens a .so/.dylib, calls aether_lib_meta, and prints a human-readable dump:

$ ae lib-info build/libscript.so
Aether Library: build/libscript.so
  Schema:        1.0
  Aether:        0.134.0-dev
  Source:        script.ae
  Functions:     3

  - aether_script_handle(ptr, ptr, ptr) -> void
        @ script.ae:3
  - double_int(int) -> int
        c_symbol: aether_double_int
        @ script.ae:7
  - greet(string) -> string
        c_symbol: aether_greet
        @ script.ae:11

The c_symbol: line is suppressed when the symbol equals aether_<name> (the default for plain exports, printing it would just be noise). @c_callback-marked functions whose Aether name is the C symbol show no c_symbol: line either; they are their own export.

Windows is a follow-up, ae lib-info returns 1 with a "DLL hosting is a follow-up" message; the metadata struct is still emitted into the DLL's .rodata and is reachable via LoadLibrary + GetProcAddress("aether_lib_meta") from any host.

What this enables for FFI hosts

Any embedder can build a typed binding without re-parsing the source:

import ctypes
lib = ctypes.CDLL("./libscript.so")
class AetherLibFunction(ctypes.Structure):
    _fields_ = [("aether_name", ctypes.c_char_p),
                ("c_symbol",    ctypes.c_char_p),
                ("signature",   ctypes.c_char_p),
                ("source_file", ctypes.c_char_p),
                ("source_line", ctypes.c_int)]
class AetherLibMeta(ctypes.Structure):
    _fields_ = [("schema_version", ctypes.c_char_p),
                ("aether_version", ctypes.c_char_p),
                ("primary_source", ctypes.c_char_p),
                ("function_count", ctypes.c_int),
                ("functions",      ctypes.POINTER(AetherLibFunction)),
                ("closure_count",  ctypes.c_int),
                ("closures",       ctypes.c_void_p)]
lib.aether_lib_meta.restype = ctypes.POINTER(AetherLibMeta)
meta = lib.aether_lib_meta()[0]
for i in range(meta.function_count):
    fn = meta.functions[i]
    print(fn.aether_name.decode(), fn.signature.decode())

Same struct walks from Java Panama, Ruby Fiddle, Go cgo. The flat-C layout is the lowest common denominator across every FFI in the wild.

Working tests

The integration suite under tests/integration/ covers:

Run them with the standard make test-ae or individually:

tests/integration/emit_lib_composite/test_emit_lib_composite.sh