Evaluation

Practical reporting on AI systems, tools, and engineering execution.

PromptForge (FlowForge-Optima) — an opinionated self‑tuning optimizer for prompts, chains, and retrieval

PromptForge (FlowForge-Optima) is an orchestration layer that automates prompt rewriting, evaluation, and versioning with deterministic replays, cost-aware objectives, and pluggable model adapters, per its README.

Git repo

October 13, 2026 2m

NeMo Gym: Production‑Ready Environment Framework for Agent Evaluation and Training

NeMo Gym provides modular, scalable environments and verification for evaluating AI agents at production scale.

Git repo

October 12, 2026 2m

Netdata: per‑second, edge‑first monitoring with on‑node ML and low overhead

Netdata is an open-source, edge-focused observability stack that captures per-second metrics, trains unsupervised anomaly models on agents, and centralizes alerts/UI without mandating metric storage in a single backend.

Git repo

October 12, 2026 2m

KubeIntellect — a human‑governed LLM orchestrator for Kubernetes diagnostics and guarded actions

KubeIntellect lets you ask a live Kubernetes cluster questions in plain English, correlates kubectl/Prometheus/Loki evidence via agent orchestration, and pauses destructive actions for explicit human approval.

Git repo

October 12, 2026 2m

Preflight — local-first observability for AI coding assistants (cost, efficiency, anti-patterns)

Preflight captures actions, cost, and efficiency for AI coding assistants with a local live dashboard and optional New Relic export for team rollups, alerts, and historical analysis.

Git repo

October 12, 2026 2m

agents-cli — CLI and skills for building, evaluating, and deploying agents on the Gemini platform

A CLI and skill bundle that equips coding assistants to scaffold, test, deploy, and register agents on Google Cloud and Gemini, with local eval tooling, CI/CD scaffolds, and observability integrations.

Git repo

October 12, 2026 2m

tsml-eval: packaged evaluation and benchmarking tools for time-series ML (v0.7.0)

tsml-eval is an open-source toolkit (v0.7.0) for benchmarking and evaluating time-series machine learning algorithms, distributed via PyPI and conda and documented on ReadTheDocs.

Git repo

October 12, 2026 2m

Inspect-Hawk (METR/hawk) — cloud-native runner for Inspect AI evaluations

A Kubernetes-native platform that orchestrates Inspect AI evaluation grids in AWS, handling pods, LLM credentials, logs, and a Postgres warehouse with a CLI and web UI.

Git repo

October 12, 2026 2m

Morphik Core — open-source multimodal retrieval engine for visually rich documents

Morphik Core is an open-source retrieval engine that claims end-to-end multimodal ingestion, search, and metadata extraction for visually rich documents, available as a hosted service and a self-hostable stack.

Git repo

October 12, 2026 2m

Arconia: Spring Boot add-on for dev services, OpenTelemetry, AI observability, and document processing

Arconia is an add-on framework for Spring Boot that bundles dev services, OpenTelemetry integration, AI observability conventions, document extraction, multitenancy helpers, and Spring Data extensions.