Get Your Model Tracked

Send your model's inputs and outputs to HumanJudge. Real humans review them and give you pass/flag verdicts — you see it all in your dashboard.

Watch first (5 min)

Kamrul Hasan

Developer Advocate · HumanJudge

The 4 steps

Getting reviewed on HumanJudge is four decisions, each one covered below.

Step 1

Account + model + API key.

Step 2

Hook it up

Python · Node · endpoint · no-code.

Step 3

Pick eval mode

Public arena or your own project.

Step 4

Configure the observer

How the signal comes back.

Step 1

Register your account and model

Sign in at humanjudge.com/developer-hub as a builder, then create a model. You'll get two things you need for every other step:

A

Project ID (a UUID)

Identifies which model the traces belong to. Shown on the model page.
B

API key

Starts with gj_sk_live_. Copy it once — you won't see it again.

Step 2

Hook your model up

Four ways to send us your model's traces. Click one to expand — the others hide.

Best when your model runs inside a Python service and you want to wrap existing LLM calls. Works with OpenAI, Anthropic, LiteLLM, LangChain, or your own model.

1. Install

shell

pip install grandjury

2. Send a trace after every LLM call

Python

import grandjury
from openai import OpenAI

gj = grandjury.GrandJury(
    api_key="gj_sk_live_...",       # from your project dashboard
    project_id="00000000-0000-0000-0000-000000000000",
)

client = OpenAI()

def answer(user_prompt: str) -> str:
    resp = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": user_prompt}],
    )
    output = resp.choices[0].message.content

    # ⬇ one line — trace is queued and sent asynchronously
    gj.trace(
        name="answer",
        input=user_prompt,
        output=output,
        model="gpt-4o",
    )
    return output

Or use the decorator (cleaner)

Python

@gj.observe(name="answer", model="gpt-4o")
def answer(user_prompt: str) -> str:
    resp = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": user_prompt}],
    )
    return resp.choices[0].message.content

# input = user_prompt, output = return value — all captured automatically

Async supported via await gj.atrace(...). Context manager via with gj.span(...). Never crashes your app — errors log silently.

Best when your model runs behind a Node backend — Next.js API routes, Express, Nest, Cloudflare Workers, or a serverless function.

1. Install

shell

npm install @grandjury/sdk

2. Send a trace after every LLM call

TypeScript

import GrandJury from "@grandjury/sdk";
import OpenAI from "openai";

const gj = new GrandJury({
  apiKey: "gj_sk_live_...",       // from your project dashboard
  projectId: "00000000-0000-0000-0000-000000000000",
});

const client = new OpenAI();

export async function answer(userPrompt: string): Promise<string> {
  const resp = await client.chat.completions.create({
    model: "gpt-4o",
    messages: [{ role: "user", content: userPrompt }],
  });
  const output = resp.choices[0].message.content ?? "";

  // ⬇ one call — trace queued and sent async, never throws
  await gj.trace({
    name: "answer",
    input: userPrompt,
    output,
    model: "gpt-4o",
  });

  return output;
}

Or wrap the whole function

TypeScript

const trackedAnswer = gj.observe(answer, {
  name: "answer",
  model: "gpt-4o",
});

// Use it exactly like the original — trace is captured automatically
const reply = await trackedAnswer(userPrompt);

Node.js 18+ (uses native fetch). Zero runtime dependencies. Errors log to console.error — your app never crashes.

Best when your model already exposes an HTTP endpoint (a hosted inference URL, an internal RAG service, a chat API). You register it once, enroll in a benchmark, and we invoke your endpoint per question — no SDK on your side.

How the request works

We POST your endpoint with a JSON body you define (request_template).
The benchmark question is substituted into {{prompt}} in your template.
Auth header is always Authorization: Bearer <your apiKey>.
We extract the AI response from your JSON reply using a JMESPath expression (response_path).

1. Register your endpoint

In your project dashboard → open your Model → Connections → paste the fields below. Or via API:

shell — for an OpenAI-compatible endpoint

curl -X PUT https://api.humanjudge.com/api/v1/models/{model_id}/endpoint-config \
  -H "Authorization: Bearer <your platform JWT>" \
  -H "Content-Type: application/json" \
  -d '{
    "endpoint": "https://your-app.com/v1/chat/completions",
    "apiKey": "sk-your-model-api-key",
    "request_template": "{\"model\":\"gpt-4o\",\"messages\":[{\"role\":\"user\",\"content\":\"{{prompt}}\"}]}",
    "response_path": "choices[0].message.content"
  }'

shell — for a custom endpoint returning {"answer": ...}

curl -X PUT https://api.humanjudge.com/api/v1/models/{model_id}/endpoint-config \
  -H "Authorization: Bearer <your platform JWT>" \
  -H "Content-Type: application/json" \
  -d '{
    "endpoint": "https://your-app.com/generate",
    "apiKey": "your-internal-token",
    "request_template": "{\"prompt\":\"{{prompt}}\"}",
    "response_path": "answer"
  }'

2. Field reference

Field	Description
`endpoint`	Full URL we POST to. Public URL required.
`apiKey`	Sent as `Authorization: Bearer <apiKey>`. If your endpoint doesn't need auth, pass an empty string.
`request_template`	A JSON string (as a string!) with `{{prompt}}` where the question goes. We substitute, then parse into a JSON body.
`response_path`	JMESPath to the AI text in your JSON response. Examples: `"answer"`, `"choices[0].message.content"`, `"data.output"`.

 3. Test before enrolling
 
The dashboard's Test endpoint button sends a sample prompt and shows what we extract. Or via API:
   shell 
 curl -X POST https://api.humanjudge.com/api/v1/models/{model_id}/test-endpoint \
  -H "Authorization: Bearer <your platform JWT>" \
  -H "Content-Type: application/json" \
  -d '{ ...same body as save... }'

# Returns { "success": true, "output": "<what we extracted>" }
# or     { "success": false, "error": "<what went wrong>" }
 
 4. Enroll in a benchmark
 
Pick one from humanjudge.com/reviewer-hub. We POST each benchmark question to your endpoint — every response becomes a trace, and reviewers grade them.

Best when your stack isn't Python or Node — you're building on a workflow tool like Zapier, n8n, Make, Retool, or shelling out from Go / Rust / Ruby / a bash script. One POST per LLM call, no dependencies.

Send a trace with a plain HTTP request

HTTP

POST https://api.humanjudge.com/api/v1/traces/ingest
Authorization: Bearer gj_sk_live_...
Content-Type: application/json

{
  "gj_inference_id": "gj_inf_1720000000_abc123",
  "name": "answer",
  "input": "What are the mental health effects of overuse in ChatGPT?",
  "output": "Overuse can lead to...",
  "model": "gpt-4o",
  "latency_ms": 1230,
  "prompt_tokens": 42,
  "completion_tokens": 128,
  "metadata": {"user_id": "u_42", "region": "us-east"}
}

Wire it into your workflow tool

Every workflow tool has an "HTTP Request" node. Wire it up like this:

Node setting	Value
Method	`POST`
URL	`https://api.humanjudge.com/api/v1/traces/ingest`
Header	`Authorization: Bearer gj_sk_live_...`
Body type	JSON
Body	The JSON object above. Reference upstream nodes with the tool's expression syntax — e.g. n8n: `{{ $json.prompt }}` · Zapier: `{{step.user_prompt}}` · Make: `{{1.input}}`.

 Field reference
     Field  Required  Notes  
 
   gj_inference_id  yes  Your unique ID for the call. Retries with the same ID are safe (idempotent).  
  input  recommended  The prompt the user gave you (string or JSON string).  
  output  recommended  What your model returned.  
  name  no  Operation label, e.g. "cover_letter".  
  model  no  Model identifier, e.g. "gpt-4o".  
  latency_ms  no  Call duration in ms.  
  prompt_tokens, completion_tokens  no  Usage stats — shows in your analytics.  
  metadata  no  Any JSON object. Use for user IDs, region tags, A/B keys.  
 
 
 
 
Response is 201 Created on new trace, 200 OK on idempotent replay. Auth failures return 401.

Field	Required	Notes
`gj_inference_id`	yes	Your unique ID for the call. Retries with the same ID are safe (idempotent).
`input`	recommended	The prompt the user gave you (string or JSON string).
`output`	recommended	What your model returned.
`name`	no	Operation label, e.g. `"cover_letter"`.
`model`	no	Model identifier, e.g. `"gpt-4o"`.
`latency_ms`	no	Call duration in ms.
`prompt_tokens`, `completion_tokens`	no	Usage stats — shows in your analytics.
`metadata`	no	Any JSON object. Use for user IDs, region tags, A/B keys.

     Step 3
 Pick your evaluation mode
 
What kind of feedback loop do you want? Two options, and you can run both on the same model.
    🏆 Arena
 
 
Enroll your model in a public benchmark. Shared questions across every enrolled model, blind reviewers, public leaderboard placement. Great for showing trust externally.
 
Best for: public trust factor, marketing proof, apples-to-apples comparison.
 Arena docs → 
   🔒 Standalone
 
 
Your own project with your own prompts. Only your model gets graded. Not on the public leaderboard. You control who sees the results.
 
Best for: internal QA, regression tracking, prompt tuning, private-domain models.
 Configured on the model page — no separate docs page yet. 
 
 
     Step 4
 Configure how the signal comes back
 
Once verdicts start landing on your traces, you need somewhere to see them. Today the observer is the dashboard; more channels are coming.
    📊 Dashboard (available now)
 
  developer-hub → your model → Traces, Analytics, Enrollments. Time-decayed karma, pass rate, per-trace reviewer feedback, flag categories.
 
   🔔 Alert channels (coming soon)
 
 
Get a webhook / Slack / Discord / email ping when your karma drops, a new flag lands, or a threshold trips. Building on our Spectator alerts engine — dedicated builder docs will land here.
 
 
 
     What happens next
   1  Your traces appear in the dashboard
  humanjudge.com/developer-hub → your project → Traces.
 
 
  2  Reviewers grade them
 
Verified human reviewers give Pass or Flag verdicts, with feedback. Blind — they don't know it's your model.
 
 
  3  You see the score
 
Analytics → time-decayed karma, pass rate, flag categories, per-trace feedback. Public leaderboard placement if you enroll in a benchmark.
 
 
 
 
     Next steps
   Create an Arena →
 Build a custom benchmark with your own questions.
   Authentication deep-dive →
 API keys vs Personal Access Tokens.
   Read evaluation data →
 Query votes, leaderboards, and reviewer feedback from Python.
   How scoring works →
 Time-decay karma, reviewer weighting, pass-rate math.