AM Consulting Open-Source Briefing · Skills Platform
AM Consulting rook mark
AM Consulting — Technology Briefing

Managing AI Skills
Across Domains

Two goals — domain-scoped skill profiles and a CI/CD pipeline that tests skills before they ship — mapped to the open-source projects that get you there. No single tool does both; the winning move is to compose two proven layers on one open standard.

SKILL.md open standard Profiles & marketplaces Eval-gated CI/CD Portable across agents

The two goals

Both reduce to the same unit — a skill: a folder with a SKILL.md instruction file plus optional scripts. Group them one way, gate them the other.

Goal 01

Domain profiles

Consolidate skills into selectable sets so a marketing team loads marketing skills without finance coming along for the ride. In the ecosystem this unit is a plugin (a bundle of skills, MCP servers, commands and hooks), served from a marketplace — a marketplace.json catalog that points at skill repos. A plugin is a domain profile.

What you need: a self-hostable catalog, per-domain bundling, and clean install/uninstall so teams see only their set.
Primary layer → registry / marketplace
Goal 02

CI/CD for skills

A skill is mostly a prompt plus optional code, so it is gated the same way software is: every change opens a PR, an evaluation suite runs assertions against it, and a failing skill is blocked from merging. Only vetted skills reach the marketplace teams pull from.

What you need: declarative test cases, pass/fail gates wired into GitHub/GitLab, plus red-team checks for prompt-injection since skills can execute code.
Primary layer → eval & test harness

Layer 1 — Profiles & marketplaces

Projects that catalog, bundle and distribute skills. The first two are infrastructure you self-host; the last two are domain-sliced content to seed your profiles.

Skill Marketplace

github.com/dukelyuu/skills-marketplace
Best fit
Bills itself as the first open-source marketplace purpose-built for agent skills — an "npm for skills." Browse a curated catalog, inspect each SKILL.md before installing, one-click import into your agent, and manage the GitHub sources that feed the registry.
◆ registry backbone◆ self-hostable◆ multi-agent
DoesCatalog + inspect + install; curates skill sources into one browsable index
AgentsClaude Code, Kiro, Cursor, Windsurf, GitHub Copilot, Cline & more
Use forThe closest thing to a ready registry backbone to fork for internal profiles

Netresearch Marketplace

github.com/netresearch/claude-code-marketplace
Pattern
A clean reference for the mechanism you would self-host: a single marketplace.json that references individual skill repos by source, so the client fetches each skill directly from its own repo on install. Built on the open agentskills.io standard.
◆ source-ref catalog◆ open standard◆ 30+ agents
DoesThin catalog-of-pointers; each skill stays in its own versioned repo
PortableClaude Code, Cursor, Copilot, Codex, Gemini CLI + 30 more
Use forThe template for your own internal marketplace — one plugin per domain

wshobson / agents

github.com/wshobson/agents
Content
A multi-harness agentic plugin marketplace: production-ready building blocks authored once in Markdown and consumed natively across many clients. Large, actively maintained, and already organized into plugins you can lift into profiles.
90 plugins199 agents161 skills106 commands
DoesSingle Markdown source → consumed by Codex, Cursor, OpenCode, Gemini, Copilot
StrengthProof that "author once, run many harnesses" scales in practice
Use forBattle-tested building blocks to seed engineering-heavy profiles

alirezarezvani / claude-skills

github.com/alirezarezvani/claude-skills
Content
A broad, explicitly domain-sliced collection — the closest existing library to your "marketing not bundled with finance" goal, already partitioned by business function rather than by tool. A strong starting inventory for non-engineering profiles.
345 skills/plugins30+ agents70+ commands8+ agent tools
DomainsMarketing, finance & commercial, product, compliance, C-level advisory, research, ops
AgentsClaude Code, Codex, Gemini CLI, Cursor + 8 more
Use forOff-the-shelf domain profiles to adapt to a client's institutional truth

Layer 2 — CI/CD & evaluation

The mature layer, borrowed from LLM-eval tooling. These are the gate that stands between a proposed skill change and the marketplace teams pull from.

promptfoo

github.com/promptfoo/promptfoo
Best fit
A developer-first, open-source eval framework: write declarative YAML test cases with assertions, run them locally or in CI, and fail the build when a skill regresses. Ships a native GitHub Action plus CLI hooks for GitLab CI, Jenkins and others — and a red-team / prompt-injection scanner, which matters the moment skills can run code.
◆ declarative YAML◆ GitHub Action◆ red-teaming◆ used by OpenAI & Anthropic
DoesAssertion-based pass/fail gates on prompts, agents & RAG; model A/B compare
CIGitHub Action, GitLab, Jenkins; blocks the merge on failure
Use forThe default merge gate — lowest-friction path to "tested before update"

DeepEval

github.com/confident-ai/deepeval
Eval
A pytest-native evaluation framework (Apache-2.0) exposing 50+ research-backed metrics — faithfulness, hallucination, relevance, bias, toxicity and more — so skill tests live right alongside ordinary unit tests and gate the same pipeline. Conversation simulation covers multi-turn scenarios, not just single-shot checks.
Apache-2.050+ metricspytest-nativemulti-turn sim
DoesMetric-driven asserts in Python/pytest; component & end-to-end eval
FitsTeams that already think in pytest and want metrics beyond pass/fail
Use forDeeper quality metrics where a binary assertion is too coarse

LangSmith

docs.langchain.com · LangSmith + LangGraph
Observability
Observability plus offline evaluations (AgentEvals / OpenEvals) wired into a CI/CD pipeline, paired with the open-source LangGraph for orchestrating agents. Heavier than a pure gate — worth it when skills are steps inside larger, multi-turn agent flows you also need to trace in production.
◆ tracing + eval◆ LangGraph OSS◆ CI pipeline
DoesOffline eval + production tracing for agent workflows
NotePlatform is managed/hosted; LangGraph orchestration is open source
Use forWhen skills are agent steps that also need production observability

LangWatch

langwatch.ai
Observability
Positions itself as a LangSmith alternative that does not require the LangChain ecosystem. Its differentiator is agent-simulation testing — evaluating multi-turn workflows end-to-end rather than isolated input/output pairs, which is closer to how a real skill behaves in a conversation.
◆ framework-agnostic◆ agent simulation◆ multi-turn
DoesEnd-to-end simulation testing of agent/skill workflows
FitsStacks that want observability without adopting LangChain
Use forFramework-neutral multi-turn skill testing

How the two layers compose

No turnkey product spans both goals — but the pieces snap together into one loop. Author skills, gate them in CI, publish the passing ones as per-domain profiles teams install selectively.

01

Author

A skill = a SKILL.md folder in your internal skills repo.

→
02

Gate in CI

PR opens → promptfoo / DeepEval runs → a failing skill can't merge.

→
03

Publish

Merge updates your marketplace.json catalog.

→
04

Bundle by domain

marketing · finance · legal — one plugin each.

→
05

Install selectively

Each team pulls only the profiles it needs.

Self-host the marketplace (fork Netresearch's pattern) · gate on merge (promptfoo) · organize into per-domain plugins.

The honest read

What the ecosystem gives you today, and where the gap — and the opportunity — actually is.

The gap

  • No single open-source project delivers both profiles and CI/CD gating end to end
  • Marketplaces catalog and distribute — they do not test what they list
  • Eval frameworks test — they have no concept of domain profiles or distribution
  • Governance (who may publish a finance skill, approvals, per-domain access) is a build-it-yourself layer
  • Prompt injection through skills that execute code is a live risk the tooling only partly covers

The play

  • Standardize on the open SKILL.md format so nothing is locked to one client
  • Fork a marketplace pattern to self-host per-domain profiles
  • Wire promptfoo as the mandatory merge gate — no untested skill ships
  • Add DeepEval where binary pass/fail is too blunt for the metric you care about
  • Own the governance layer yourself — it is exactly where a consultancy adds durable value
“The model is rarely the hard part — the system around it is.”
Avishay Meron
Avishay Meron
CEO & Founder · AM Consulting