DatEngine 🚀 Modular AI Agentic Framework written in Nim lang

nimble install datengine

API reference

DatEngine is an app-agnostic agentic engine written in Nim. Made to build self-hosted, AI agents via command line and browsers. Designed as a pure library with no HTTP server or UI. A CLI, REST, or WebSocket adapter drives it.

The engine orchestrates an LLM agent loop, manages sessions, executes tools with a strict safety envelope, and automates browsers via CDP (Chrome DevTools). Every wire format (config, sessions, tool schemas) flows through openparser as typed Nim objects.

• Any OpenAI-compatible endpoints
• Streaming SSE: token-by-token deltas via ChaChaChat's async HTTP client
• Tool calling: agentic function calling with typed parameter schemas and automatic argument decoding
• Session persistence: Boogie RDBMS with indexed columns, conversation history survives restarts
• Truncation: configurable history window with system-message preservation via replaceMessages
• Cancellation: cooperative cancellation via agent flag (checked between tool iterations)
• Skills: markdown files with YAML frontmatter, fuzzy-matched per turn via floof and injected into the system message (see below)
• Two opt-in sources: global (globalFs disk skills at ~/.<myagent>/skills) and per-session workspace (<skillsDir> relative to Workspace.root); workspace overrides global when names collide. Both empty = skills disabled.
• floof fuzzy matching: SIMD-accelerated subsequence search of user input against keywords and names; skillMinScore threshold (default 0.5), top-N cap via skillMaxPerTurn (default 3).
• flysystem loading: skills are read through flysystem drivers (globalFs.disk("skills") + Workspace.fs.disk("workspace")); newSkillRegistryFromDrivers does driver-level listing, gitignore rules apply only to workspace skills.
• Model-facing tools: skill_list and skill_read let the LLM browse skill content explicitly.

Phase 1 — global init (once at startup): all dirty wiring inside initDatEngine, providers auto-synced:

import datengine

# single call with large param set; everything dirty happens inside:
# newGlobalFs (skills/config/providers at ~/.myagent) + newProviderStore
# + auto syncFromGlobalFs (YAML/JSON at ~/.myagent/providers/*.yml)
# + newSessionStore + baseDir derivation + ensure dirs
var engee = initDatEngine(
  globalHome = getHomeDir() / ".myagent",
  baseDir = "./storage",
  mode = amBuild,              # ask (default, readOnly) / plan / build
  skillsDir = "skills",
  maxIterations = 10,       # max tool-loop iterations per turn (chachachat runToolLoop)
  truncateTo = 50           # keep last N non-system messages in history; 0 = keep all
)
# or from YAML: let cfg = loadEngineConfig("engine.yml"); var engee = initDatEngine(cfg)

# providers via global single source at ~/.myagent/providers (add via API, not init):
engee.addProvider("openai", "https://api.openai.com/v1", "gpt-4o", apiKeyEnv="OPENAI_API_KEY")
# name unique: openai -> openai-1/-2 on collision
# users can also add ~/.myagent/providers/ollama.json manually:
# {"name":"ollama","baseUrl":"http://localhost:11434/v1","model":"llama3"}
echo engee.listProviders().len

# model discovery (async, cached in ProviderStore under models:<name>):
import std/asyncdispatch
let models = waitFor engee.fetchProviderModels("openai")          # uses stored baseUrl/apiKeyEnv
# or before provider exists: let models = waitFor engee.fetchProviderModelsForUrl("https://opencode.ai/zen/go/v1", "", "")
# pick and update: var cfg = engee.getProvider("openai").get; cfg.model = models[0].id; engee.upsertProvider(cfg)
# cached access: let cached = engee.getProviderModels("openai")
echo models.len

Phase 2 — per workspace / per agent (per session / per request):

# isolated Workspace at ./storage/workspaces/<sessionId> + artifacts, gitignore-filtered
let agent = engee.newAgent(sessionId) # or engee.newAgentForUser(sessionId, userId)
# mode-aware tools + skills are auto-wired inside (Ask registers no fs_tools)
let resp = waitFor agent.run("Analyze the files in this workspace")
echo resp.text

# runtime mode switch
engee.setMode(amPlan)              # affects next workspaces
agent.setMode(amPlan)              # affects this agent's workspace (re-applies PolicyRules)
let ws = engee.getWorkspaceForSession(sessionId)
echo ws.root # ./storage/workspaces/<id>

• Allowlisted CLI Binary allowlist, no shell metacharacters, cwd confinement (Workspace.root), per-tool output caps and timeouts

• rtk proxy Token-optimized output for the model (ls, tree, read, grep, find, diff, wc, json)

• Document extraction: pdftotext, pdfinfo, pdftoppm, pdftohtml, vips, sips, convert, ffmpeg; renders/screenshots land on the per-session artifacts disk (Workspace.artifactWrite)

• Per-session gitignore-aware workspace Workspace owns a per-session Filesystem with disks workspace (filtered via pkg/gitignore IgnoreStack; .env, .git, node_modules never reach the model) + artifacts (unfiltered sibling for downloads/renders). Every path is LocalDriver.resolvePath traversal-proof and atomic. Global state lives on a separate host-wide globalFs: Filesystem with named disks skills/config at ~/.myagent. AgentMode (ask/plan/build) enforces flysystem PolicyRules (ask/plan = readOnly=true, build = writable with 10 MB caps).

• Browser automation chopchop CDP: goto, waitForNavigation(NetworkIdle), querySelector, evaluate, screenshot, click, typeText

• Safety envelope per-tool byte/line caps, timeouts (30/60/120s), truncation markers, process kill on timeout, plus PolicyError on AgentMode violations

• Per-session todos Session.todos: seq[TodoItem] (id, content, status: pending|in_progress|completed|cancelled, priority: high|medium|low) persisted via SessionStore (sessions.todosJson). LLM tools todo_create(content, priority?) → id, todo_update(id, content?, status?, priority?) (single-item patch by id), todo_delete(id), todo_read (explicit, not auto-injected). Enforced “plan before build”: in amPlan/amBuild any non-todo tool is blocked until at least one todo exists (ask exempt). Managed per-session, visible across newAgent(sessionId) reloads.

• OpenAPI-compatible: ProviderConfig(name, baseUrl, model, apiKey, apiKeyEnv) — name globally unique, collisions auto-suffixed blabla → blabla-1/blabla-2.
• Global single source: globalFs disk providers at ~/.myagent/providers (flysystem, YAML/JSON via openparser), backed by boogie DocumentStore at ~/.myagent/providers.ddb/.wal. Available everywhere, all sessions/workspaces.
• API: newProviderStore(home, globalFs), syncFromGlobalFs(), listProviders(), getProvider(name), upsertProvider / upsertProviderUnique, deleteProvider, resolveApiKey. YAML example at providers/openai.yml: name: openai + baseUrl: "https://api.openai.com/v1" (quote URLs) + apiKeyEnv.
• Model discovery: OpenAI-compatible GET {baseUrl}/models → {"object":"list","data":[{"id","created","owned_by"}]} mapped via openparser fromJson into LLModel (models.nim). Async fetcher fetchProviderModels(baseUrl, apiKey) / fetchProviderModels(cfg) / fetchProviderModelsForUrl(baseUrl, apiKey, apiKeyEnv) — no auto-discover on addProvider; caller does let models = await engee.fetchProviderModels("openai") or await engee.fetchProviderModelsForUrl(baseUrl, apiKey, apiKeyEnv) once when adding a new provider, then picks model for addProvider. Fetched list is cached in ProviderStore under models:<name> (ModelListResponse wrapper) via setProviderModels/getProviderModels/hasProviderModels; refresh requires explicit fetchProviderModels call. DatEngine wrappers: fetchProviderModels(providerName), fetchProviderModels(cfg), fetchProviderModelsForUrl, getProviderModels, hasProviderModels. Live example: https://opencode.ai/zen/go/v1/models returns mimo-v2.5, kimi-k2.5, glm-5.3, etc.

• Global, tools-only, app-controlled: ~/.myagent/plugins/*.so|.dylib|.dll (host-wide globalFs disk plugins at newGlobalFs). When enabled, DatEngine owns a PluginManager (pluginkit ABI 1, semver, NimVersion gate). Plugins are dynamic libraries built via nim c --app:lib --mm:orc --threads:on myplugin.nim.
• Contract: plugin exports plugin_datengine_tools_json*(): cstring {.exportc, cdecl, dynlib.} → JSON array [{"name","description","schema":{…}}] and per-tool handler plugin_tool_<name>*(argsJson: cstring): cstring {.exportc, cdecl, dynlib.} (sync, args is JSON object string). Host at src/datengine/plugins.nim:60 attachPluginTools(manager, registry) per-session (in engine.bindAgent) discovers plugin_datengine_tools_json via dynlib.symAddr, parses JSON, and registers each as Tool with a chachachat.ToolHandler wrapper that forwards args: JsonNode as $args cstring and returns handler result. Duplicate names skipped (first wins).
• Lifecycle: app drives via DatEngine public API — getPluginsDir, listInstalledPlugins (walk home/plugins), loadPlugin(path) → id, activatePlugin(id), unloadPlugin(id), installPlugin(srcPath, destName?) → dest (copy to home/plugins), uninstallPlugin(id) (unload then removeFile), listLoadedPlugins, hasPlugin, getPluginManager. DatEngine.close unloads all. No auto-scan at initDatEngine, no watcher, no permission enforcement (manifest permissions at pluginkit.nim:94 ignored for now). Example at packages/supranim-packages/pluginkit/example/helloworld.nim style plus plugin_datengine_tools_json/plugin_tool_my_echo.
• Config: AgentConfig.pluginsDir*: string (global, empty → home/plugins), YAML-friendly, default via home/plugins at engine.initDatEngine and ensurePluginsDir.
• Security: ABI PluginAbiVersion=1 check at pluginkit.nim:540, version/ NimVersion gates. Tools run with same safety envelope as host tools; ask/plan read-only policies still apply via AgentMode.

Plugins are global dynamic libraries that extend the LLM tool surface at runtime. The host owns the lifecycle; there is no auto-scan or watcher. Build with nim c --app:lib --mm:orc --threads:on.

Plugin side (myplugin.nim):

import pkg/pluginkit
import std/json

plugin myplugin, {
  name: "MyPlugin",
  author: "Example",
  description: "Echo text",
  license: "MIT",
  url: "https://example.com",
  version: "0.1.0"
}:
  discard

proc plugin_datengine_tools_json*(): cstring {.exportc, cdecl, dynlib.} =
  ## JSON array of tools: host at src/datengine/plugins.nim:60 discovers this symbol
  """[{"name":"my_echo","description":"Echo text","schema":{"type":"object","properties":{"text":{"type":"string"}},"required":["text"]}}]"""

proc plugin_tool_my_echo*(argsJson: cstring): cstring {.exportc, cdecl, dynlib.} =
  ## Per-tool handler: `plugin_tool_<name>` — argsJson is the tool args as JSON object string
  let args = parseJson($argsJson)
  let text = args{"text"}.getStr("")
  cstring("echo:" & text)

Compile:

nim c --app:lib --mm:orc --threads:on -o:myplugin.dylib myplugin.nim
# Linux: .so, macOS: .dylib, Windows: .dll

Host side (datengine):

import datengine
import std/asyncdispatch

let engee = initDatEngine(
  globalHome = getHomeDir() / ".myagent",
  baseDir = "./storage",
  mode = amBuild
)
discard engee.addProvider("openai", "https://api.openai.com/v1", "gpt-4o", apiKeyEnv="OPENAI_API_KEY")

# install persists to ~/.myagent/plugins/ (or custom AgentConfig.pluginsDir)
let dest = engee.installPlugin("/path/to/myplugin.dylib")
# or: let dest = engee.installPlugin("/path/to/myplugin.dylib", "myplugin.dylib")

# load + activate (app-controlled, no auto-load)
let id = engee.loadPlugin(dest)       # → hash id, checks ABI/semver/NimVersion at pluginkit.nim:515
engee.activatePlugin(id)              # calls plugin_init (NimMain), status → pluginStatusActive

# discovery
echo engee.getPluginsDir()            # ~/.myagent/plugins or custom
echo engee.listInstalledPlugins()     # ["…/plugins/myplugin.dylib"]
echo engee.listLoadedPlugins().len    # 1
echo engee.hasPlugin(id)              # true

# per-session attach: engine.bindAgent attaches currently loaded plugins to each newAgent
let agent = engee.newAgent("sess-1")
assert agent.registry.hasTool("my_echo")
assert agent.registry.hasTool("todo_create") # built-ins remain
let res = waitFor agent.registry.getTool("my_echo").get.handler("my_echo", %*{"text":"hello"})
echo res # echo:hello

# unload / uninstall (app-controlled)
engee.unloadPlugin(id)
assert not engee.hasPlugin(id)
engee.uninstallPlugin(id) # not needed if already unloaded: no-op; otherwise unloads then removeFile
# or after re-load:
# let id2 = engee.loadPlugin(dest); engee.activatePlugin(id2); engee.uninstallPlugin(id2)
# engee.close() unloads all remaining plugins

See src/datengine/tools/plugin.nim:1 shim and src/datengine/plugins.nim:60 attachPluginTools.

• Boogie RDBMS https://github.com/openpeeps/boogie
  Indexed relational store for sessions (indexed columns, structured queries)

• Boogie DocumentStore https://github.com/openpeeps/boogie
  Schemaless JSON store for providers (providers docstore, putObj/getObj, pairs), single global file.

• Flysystem https://github.com/openpeeps/flysystem
  Multi-disk sandbox: per-session Workspace.fs (workspace + artifacts disks, traversal-proof, atomic writes) + host-wide globalFs (skills + config + providers + plugins disks at ~/.myagent). All reads/writes go through StorageDriver; no raw readFile paths escape the engine.

• OpenParser https://github.com/openpeeps/openparser
  Collection parsers/dumpers: Full QR family/JSON/TOML/YAML/FBE/DotEnv/iCal/Regex/SQL/Gettext (po/mo) and more

Markdown + YAML frontmatter; the engine injects matching raw skill bodies into the system message each turn:

---
name: pdf-analysis
description: How to extract and analyze PDF documents
keywords: [pdf, extract, document, ocr]
---
# PDF Analysis
(instructions for the LLM...)

• Mock LLM: mock OpenAI-compatible server with streaming SSE and tool_call responses
• ~155 tests: core types, agent lifecycle, tool safety, session round-trip, config parsing, truncation, persistence, boogie storage, skills, providers (DocumentStore, YAML/JSON sync, unique suffix, globalFs)

Skills are opt-in via flysystem disks: host-wide globalFs (~/.myagent/skills) and per-session Workspace (<workspace>/skills). Loaded through drivers, not raw paths.

~/.myagent/                    # host globalFs (newGlobalFs)
├── skills/
│   └── pdf-analysis.md       # available in every session
└── config/
    └── engine.yml

<workspace>/                   # per-session Workspace.root via newWorkspace/forSession
└── skills/
    └── project-specific/     # <name>/SKILL.md layout also works
        └── SKILL.md

Legacy newSkillRegistry(fs, skillsDir, globalPath) and newFsTool(root) shims remain for single-workspace scripts, but web apps should use globalFs + Workspace + newSkillRegistryFromDrivers.

On each run, user input is fuzzy-matched against skill keywords and names (floof); matching raw markdown bodies are injected into the system message for that turn. The model can also call skill_list / skill_read explicitly. Agent.setMode and Workspace.setMode can be used to gate plan vs build at runtime.

src/datengine/
├── agent.nim          # Agent loop: chachachat Conversation + tools + hooks, holds Workspace, setMode
├── config.nim         # ProviderConfig(name, baseUrl, model, apiKeyEnv) — name globally unique; EngineConfig (agent only, no provider; providers via ProviderStore)
├── models.nim         # Model discovery: LLModel/ModelListResponse via openparser fromJson, async GET {baseUrl}/models (e.g. https://opencode.ai/zen/go/v1/models) + caching
├── mockllm.nim        # OpenAI-compatible mock server for testing
├── prompt.nim         # System prompt builder from tool schemas
├── providers.nim      # Global providers (DocumentStore at ~/.myagent/providers.ddb + globalFs disk providers, OpenAPI-compatible, YAML/JSON via openparser, unique suffix, syncFromGlobalFs) + model cache models:<name>
├── serialization.nim  # openparser glue: fromJsonArgs, jsonOrEmpty, helpers
├── session.nim        # Indexed relational store for sessions (indexed columns, structured queries) + per-session todos (persisted todosJson, LLM-managed via todo_*)
├── skills.nim         # Skill loading via flysystem drivers (globalFs + workspace) + floof matching
├── workspace.nim      # Per-session Workspace (flysystem Filesystem: workspace + artifacts) + globalFs (skills/config/providers at ~/.myagent), AgentMode PolicyRules, gitignore stack, per-session forSession helper
├── engine.nim         # High-level DatEngine (initDatEngine large params, auto sync providers, newAgent factory, getters/setters, fetchProviderModels async wrappers, PluginManager (global plugins at ~/.myagent/plugins), no global agent)
├── plugins.nim        # Host plugin manager (global, tools-only, app-controlled load/activate/install/uninstall, per-session attachPluginTools via plugin_datengine_tools_json → plugin_tool_<name>)
├── tools.nim          # Tool, ToolRegistry, ToolResult, JSON Schema helpers
└── tools/
    ├── cli.nim        # Allowlisted subprocess (threadpool, caps, timeouts): workdir = Workspace.root
    ├── rtk.nim        # rtk output proxy
    ├── document.nim   # poppler/vips/sips/ffmpeg wrappers: outputs to artifacts disk
    ├── fs.nim         # FsTool adapter over Workspace disk (shares LocalDriver + IgnoreStack)
    ├── todo.nim       # Per-session todos (persisted via SessionStore, id-based todo_create/update/delete/read, plan-before-build enforcement)
    ├── plugin.nim     # Thin shim for plugin authors (re-exports pluginkit, documents plugin_datengine_tools_json/plugin_tool_<name> contract)
    └── browser.nim    # chopchop CDP browser automation: screenshots to artifacts disk

c-blake/bu: ~70 Nim-native CLI tools (pipe-oriented, zero-config, faster than GNU coreutils). Tier 1 integration planned for agent toolchains:

• Core types (Tool, ToolResult, ToolRegistry, serialization)
• Provider layer (chachachat: LLMClient, SSE, streaming hooks)
• Agent loop (Conversation-backed turns, tool calling, truncation, cancellation)
• CLI/RTK tools (allowlist, threadpool subprocess, safety envelope)
• FS tools (flysystem + gitignore workspace sandbox)
• Workspace (per-session flysystem Filesystem: workspace + artifacts, host-wide globalFs skills/config, AgentMode ask/plan/build with PolicyRules, forSession helper)
• Document tools (poppler/vips/sips/ffmpeg)
• Browser tools (chopchop CDP)
• Session persistence (Boogie RDBMS store)
• Skills (marvdown frontmatter, floof fuzzy matching, global + workspace sources via flysystem drivers)
• Config (YAML parsing with defaults, AgentMode, workspace base)
• Mock LLM server (OpenAI-compatible, streaming SSE, tool_call)
• Test suite (140 tests across 7 test files)
• BU CLI tools (dups, topn, ndelta, cols, noc, ft, ...)
• Prompt caching and context window management
• Rate limiting and retry policies
• Multi-agent orchestration
• Vision pipeline (pdftoppm → vips → base64 → model)
• RAG integration (Boogie vector retrieval)
• REST/WebSocket transport layer (powpow HTTP server)
• Web UI for agent interaction

• 🐛 Found a bug? Create a new Issue
• 👋 Wanna help? Fork it!

LGPLv3 license. Made by Humans from OpenPeeps.
Copyright OpenPeeps & Contributors — All rights reserved.