kyotofin/tax-doc-classifier

Play the video in the post below

tax-doc-classifier reads each page of a tax PDF as text and sends it to Jev in one request with two questions: which of 7 page kinds it is, and which form it is, from 230 options plus “not in this list”. Five corporate and foreign forms (5471, 8865, 8933, 1118 and 5713) stand in for their 35 schedules, so a page matching one of them gets a second request to pick the exact schedule, for 261 forms in all. Nothing is trained and nothing is hosted: the classifier is a JSON file of form descriptions generated from the IRS’s own PDFs, with Jev as the decision layer.

Nakshatra Saxena’s team ingests thousands of tax documents through an LLM pipeline built the previous tax season, and this classifier is the Jev replacement for it. The eval, reproducible with pnpm eval, scores a page as an error if the answer is wrong or its confidence is below 0.95. On 314 pages of filled TaxCalcBench forms (W-2, 1040, 1099 and 1098 series) it recorded zero wrong pages and zero strict errors for $0.36. On 753 pages of blank IRS forms covering all 261 form types, none were wrong, but 38 pages (5.05%) fell below the confidence bar: instruction pages that name no form, deep pages of some corporate forms, and single-schedule forms hedging against their parent.

Against the previous classifier, which sent each page as a PDF to Claude Sonnet, the README reports $0.00115 per page instead of $0.039 and about 0.5 seconds instead of 3.3, or 34 times cheaper and 6 times faster.

How it uses Jev

export function jevBackend(opts: JevOptions = {}): Backend {
  const apiKey = opts.apiKey ?? process.env.TYPESAFE_API_KEY
  if (!apiKey) throw new Error('TYPESAFE_API_KEY is not set')
  const model = opts.model ?? 'jev-latest'
  const url = `${opts.baseUrl ?? 'https://api.typesafe.ai'}/v1/systemone`
  const timeoutMs = opts.timeoutMs ?? 90_000
  const retries = opts.retries ?? 4
  return {
    async ask(state, questions) {
      const body = JSON.stringify({ model, state, questions })
      let lastErr: unknown
      for (let attempt = 0; attempt < retries; attempt++) {
        try {
          const res = await fetch(url, {
            method: 'POST',

View in src/backend.ts

Open on GitHub