Build an Online LLM-as-a-Judge with the LangSmith CLI and a Coding Agent
From the video description
Chat LangChain gets thousands of traces a day, which is far too many to read by hand. In this walkthrough we set up an online LLM-as-a-judge in LangSmith that scores every incoming thread for user frustration, without leaving the terminal. We install the latest LangSmith skills, point a coding agent at the LangSmith CLI to create the evaluator, review the flagged threads in the UI, then update the sampling rate to control cost.
Topics
Sources
Video by LangChain, played through YouTube.
- Indie Picks
Data and selection notes
Indie Picks uses video metadata for classification. The description above is an uploader excerpt, not an independent review.
Indie Picks: Independently collected SaaS and AI product videos, organized by a founder handbook. Topics and relevance are metadata-indexed; full-video editorial review has not been completed. Descriptions are uploader excerpts where available.
Views, duration and playback status refresh separately from the published catalog. When an update fails, existing data remains visible.
Selection checks
- SaaS & AI candidates: Matches the configured criteria
- Editor-approved picks: Editorial review is required