Coding tests

A coding test is a timed set of programming problems, executed against real test cases and then scored. Use it when you need evidence that somebody can write working code, rather than talk about it. It is a separate product from interviews: nobody speaks, nothing is recorded, and the result is a score per problem plus a capability profile.

Pick it over a coding interview round when the answer has to be graded by execution. Pick the interview when you want to hear the candidate reason.

Lifecycle

assign ──> assigned ──> started ──> in_progress ──> completed ──> report
              │            │             │
              │            └─────────────┴──> timed_out
              ├──> cancelled
              └──> timed_out (never opened, 7 days)

assigned, started and in_progress are the live statuses. The other three are terminal.

Tests are authored in the product, not over the API

/v1 reads the catalogue. It does not write it. Problems, hidden test cases, scoring rubrics and expected complexities are created by an administrator in the Expert Hire portal, because publishing them over the same API a candidate's browser talks to is how an answer key leaks.

So the first step of any integration is a conversation, not a call: ask us to author the test, or author it yourself in the portal, then list it here.

The catalogue

curl -G https://prep-api.experthire.cloud/v1/coding-tests \
  -H "Authorization: Bearer $EH_SECRET_KEY" \
  --data-urlencode "difficulty=medium"

{
  "object": "list",
  "data": [
    {
      "object": "coding_test",
      "id": "0d7e2a11-5c93-4b3e-b0a2-72c9a1f5e880",
      "title": "Backend Fundamentals",
      "slug": "backend-fundamentals",
      "difficulty": "medium",
      "time_limit_minutes": 60,
      "max_attempts_per_problem": 5,
      "allowed_languages": ["python", "java", "go"],
      "created_at": 1786348200
    }
  ],
  "has_more": false,
  "total_count": 1
}

You get the tests your organization authored plus the shared Expert Hire catalogue. Filter with difficulty and skill_slug. There is no paging here: the whole catalogue comes back in one response, and has_more is always false.

Fetching one test adds problems, with one representative problem per slot:

curl https://prep-api.experthire.cloud/v1/coding-tests/$TEST_ID \
  -H "Authorization: Bearer $EH_SECRET_KEY"

A problem carries the statement, constraints, input and output format, limits, starter code and its sample cases. Hidden test cases, rubrics, reference solutions and expected complexities are stripped before the response is built.

Assigning a test

curl -X POST https://prep-api.experthire.cloud/v1/coding-sessions \
  -H "Authorization: Bearer $EH_SECRET_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "candidate": { "email": "[email protected]", "name": "Ada Lovelace" },
    "test_id": "0d7e2a11-5c93-4b3e-b0a2-72c9a1f5e880"
  }'

201:

{
  "object": "coding_session",
  "id": "5b1f7c88-0a3e-4d92-9e77-3c2d5a4b6e01",
  "status": "assigned",
  "candidate_id": "7f1c2b90-3a44-4d51-9c0e-1b2f9c4d1a55",
  "test_id": "0d7e2a11-5c93-4b3e-b0a2-72c9a1f5e880",
  "test_title": "Backend Fundamentals",
  "report_available": false,
  "resumed": false,
  "environment": "live",
  "livemode": true,
  "created_at": 1786348200
}

Pass an existing candidate_id instead of candidate if you already have one.

The clock does not start here. assigned means the session exists and the credit is spent; the time limit begins when the candidate opens it.

This endpoint does not need Idempotency-Key, though it honours one. Assigning is idempotent by candidate and test: if that candidate already holds a live session for the same test, you get 200 with resumed: true and nothing is charged. A retry after a timeout is safe.

Getting the candidate in

curl -X POST https://prep-api.experthire.cloud/v1/coding-sessions/$SESSION_ID/launch-link \
  -H "Authorization: Bearer $EH_SECRET_KEY" \
  -H "Content-Type: application/json" \
  -d '{"ttl_seconds": 604800, "redirect_url": "https://app.example.com/done"}'

{
  "object": "launch_link",
  "subject_kind": "coding",
  "subject_id": "5b1f7c88-0a3e-4d92-9e77-3c2d5a4b6e01",
  "url": "https://room.experthire.io/launch?token=ehp_lt_...",
  "token": "ehp_lt_...",
  "expires_at": 1786953000
}

Email the url. It is single use, and the token is stored hashed, so this response is the only time you see it.

While the session is still assigned, expires_at is capped to the end of the seven day assignment window measured from created_at, so it can come back earlier than the ttl_seconds you asked for. Past that window the assignment has expired and minting a link is refused with assignment_expired rather than handing you a link that opens nothing.

What the candidate sees

The hosted room opens on a pre-join screen, then an editor with the problem statement, its sample cases, a language picker, Run, Submit and a countdown. Proctoring signals are reported by the browser.

Under it is a small set of calls, all authenticated with the session token from the launch exchange rather than your secret key. Build them yourself only if you are going headless.

curl -X POST https://prep-api.experthire.cloud/v1/coding-sessions/$SESSION_ID/start \
  -H "Authorization: Bearer $EH_SESSION_TOKEN"

start stamps the clock and returns the problems this session locked, the attempt budget per problem, allowed_languages and expires_at. It is idempotent: a reload calls it again and gets the original started_at back, so reloading cannot buy extra time.

Build the language picker from allowed_languages on that response. It is the test's allowlist already narrowed to what the executor installs, so it cannot offer a language every submission would refuse. Today that set is c, cpp, csharp, go, java, javascript, php, python, ruby, rust, swift and typescript.

Run and submit are different operations

curl -X POST https://prep-api.experthire.cloud/v1/coding-sessions/$SESSION_ID/run \
  -H "Authorization: Bearer $EH_SESSION_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "problem_id": "9a3c7e50-1f22-4c8b-9d31-7e5b0a2c6d44",
    "language": "python",
    "source_code": "print(sum(map(int, input().split())))"
  }'

Run exists so the candidate can iterate without paying for it. Offer it freely. A room that hides it turns the test into a submit-only exam where every trial costs an attempt.

Send stdin to run against your own input instead of the sample cases. There is nothing to compare that output against, so every case comes back passed: false and the verdict reads WA. Read results[0].stdout and ignore the verdict on that path.

Submitting returns a receipt, not a result:

{
  "object": "coding_submission_receipt",
  "submission_id": "c2f0a9d4-88b1-4e3a-9c05-1d7f6b3e2a90",
  "attempt_number": 1,
  "remaining": 4
}

Then poll GET /v1/coding-sessions/{id}/submissions/{sid} until status leaves queued and running. The verdict is PENDING until then, and one of AC, WA, PARTIAL, TLE, MLE, RE or CE once it lands.

Hidden cases never leak

A submission is judged against every active case for the problem. The response tells the candidate how they did on the hidden ones without telling them what they are: a hidden case carries index, is_sample, passed and status, and nothing else. Sample cases additionally keep time_ms, memory_kb and stderr_excerpt. A submission's cases never carry stdout, not even for a sample case. stdout is on the run response only.

One redaction rule serves this API and the product, so the two surfaces cannot drift apart and start publishing hidden inputs.

A judge error is not a wrong answer

{ "object": "coding_submission", "status": "error", "verdict": "PENDING" }

status: "error" means the execution sandbox failed: a runtime that was not installed, a full queue, a pod that died mid batch. It is ours, not the candidate's, and it is retryable.

Two things follow from that. attempt_number keeps counting up across errored submissions, so it can exceed max_attempts; the budget is what remaining reports, not the attempt number. And remaining is max_attempts minus that same attempt number, so once a session has any errored submission it understates the real budget by one per errored row. Treat it as a floor.

Time and limits

The server owns the clock. GET /v1/coding-sessions/{id}/timer returns remaining_seconds, timed and expired. Count down locally between polls but never upward. timed is false for a test with no time limit, and remaining_seconds means nothing then.

Each execution gets the problem's time_limit_ms of CPU, plus an allowance for the language runtime's own startup so the limit means time available to the candidate's code. Both are capped at ten seconds, and a whole batch is capped at ninety.

Past the deadline, run and submit are refused with session_expired. Finalize still works and still scores the session, but a session finalized past its limit is stamped timed_out rather than completed, and the report endpoint serves only completed. The scores are on the session object either way.

Proctoring events

curl -X POST https://prep-api.experthire.cloud/v1/coding-sessions/$SESSION_ID/events \
  -H "Authorization: Bearer $EH_SESSION_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"events": [{"type": "paste", "timestamp": 1786351860}]}'

The key is type here, not event as on the interview endpoint, and each entry may carry a free-form meta object. Between 1 and 200 per request, and 200 per session in total; past that recorded comes back 0 rather than erroring. A completed or timed_out session records nothing and returns recorded: 0. A cancelled session still records, so stop reporting when you cancel.

Names are not neutral. large_paste, paste and copy carry a known penalty against the integrity signal, and any other name is penalized as well. Agree your vocabulary with us before inventing one.

Finalizing

curl -X POST https://prep-api.experthire.cloud/v1/coding-sessions/$SESSION_ID/finalize \
  -H "Authorization: Bearer $EH_SESSION_TOKEN"

{
  "object": "coding_finalization",
  "session_id": "5b1f7c88-0a3e-4d92-9e77-3c2d5a4b6e01",
  "status": "in_progress",
  "already_completed": false,
  "problems_scoring": 3,
  "report_available": false
}

Scoring runs asynchronously, so status is often still in_progress when this returns. Wait for the webhook rather than looping on the report.

It is safe to call more than once. A session that is already complete comes back with already_completed: true and nothing is rescored.

While a submission is still being judged, finalize returns 409 judging_in_progress. Retry in a few seconds. It is refusing on purpose: scoring around a live submission would drop the attempt the candidate just made.

A candidate who closes the tab and never comes back is finalized for you, as long as the test is timed. A sweeper runs every five minutes and finalizes sessions in started or in_progress past their time limit plus a thirty minute grace. Untimed tests are excluded, as noted below.

The report

curl https://prep-api.experthire.cloud/v1/coding-sessions/$SESSION_ID/report \
  -H "Authorization: Bearer $EH_SECRET_KEY"

Each problem's final_score is 50% correctness, 20% efficiency, 20% code quality and 10% complexity. Correctness and efficiency are measured from execution. Code quality and complexity come from a model reading the best submission.

The report is a judgment, not a transcript. The candidate's source code is not in it. If you need the code, read it from the submissions inside the session, which requires the candidate's session token.

The webhook

Two events fire for a coding test. Register an endpoint as described in webhooks; everything there about signatures, retries and deduplication applies unchanged.

There is no separate report event. The write that completes the session is the write that makes the report exist, so coding.completed is the only signal you need.

Nothing is emitted for a cancel, for an assign that resumed an existing session, or for an assignment that expires unopened.

The payload carries the scores, the readiness band and the sub-scores, but never the candidate's code and nothing that would reveal a hidden case.

Metering

One credit from the coding pool, taken when you assign. Check what is left on GET /v1/usage under credits.coding; the feature gate is modules.coding in the same response.

The credit is refunded in exactly one case: you cancel a session the candidate never opened.

curl -X POST https://prep-api.experthire.cloud/v1/coding-sessions/$SESSION_ID/cancel \
  -H "Authorization: Bearer $EH_SECRET_KEY"

Cancel a session that has already started and you get 400 session_already_started and no credit. Cancel one that is already cancelled and you get a 200 that changes nothing and does not refund twice.

Two cases that look like refunds and are not:

• An assignment nobody opened for seven days flips to timed_out. No credit comes back, because expiry is not a decision you made. Cancel is available for that whole window and is how you get the credit back.
• Sandbox. Sandbox usage is a monthly counter rather than a balance, so there is nothing to hand back. A sandbox cancel succeeds and restores nothing.

Reads cost nothing. Listing the catalogue, reading a session and pulling a report are never metered.

What is not here yet

Honest gaps, so you do not design around something that does not exist.

• No authoring API. You cannot create a test, a problem or a test case over /v1. Tests are authored in the product.
• No embedded room. POST /v1/sessions binds interviews only, so a coding test cannot be run in an iframe on your domain today. Hosted link or headless.
• Idempotency-Key is optional. Assigning is also idempotent by candidate and test, which covers the common retry. Nothing else on this surface is.
• No source code in the report. Only a session token can read submissions, and it expires with the session.
• No auto-finalize for untimed tests. The sweeper only picks up sessions that have a time limit. An abandoned session on an untimed test stays started or in_progress until somebody finalizes it.
• Nothing is emitted when an assignment expires unopened. The seven day sweep flips the status silently. If you track assignments, reconcile them against GET /v1/coding-sessions rather than waiting for an event.
• The session list is organization-wide. With a live key it returns every non-sandbox coding session in your organization, including ones created inside the prep product rather than by you. Match on the ids you created.