Skip to content
OnticBeta
legal

Reference Implementation: Legal Self-Help Services

Reference Implementation: Legal Self-Help Services

Case Study: Alaska AVA Probate Chatbot

Incident Overview

  • System: Alaska Virtual Assistant (AVA) - AI chatbot for probate self-help
  • Developer: Alaska Court System
  • Year: 2026 (ongoing)
  • Severity: High
  • Documented Consequence: Hallucinated nonexistent law school despite retrieval constraints. Project timeline expanded from 3 months to 1+ year; scope drastically narrowed from full probate guidance to "narrow scenario testing."

Why This Is SAF

Legal self-help services must ground advice in actual statutes, court rules, and institutional resources. AVA hallucinated a nonexistent Alaska law school when answering probate questions, demonstrating that even retrieval-augmented systems fail without structural completeness gates. State-specific procedural requirements cannot be approximated—they must be verified against canonical sources.

Domain Ontology: Legal Self-Help (Probate)

State Space Definition

{
  "domain": "Legal Services/Self-Help/Probate",
  "state_axes": [
    {
      "name": "jurisdiction",
      "description": "The state/territory with legal authority over the estate",
      "type": "categorical",
      "values": ["Alaska", "Alabama", "...", "Wyoming"],
      "required": true,
      "authority_source": "Court jurisdiction rules"
    },
    {
      "name": "form_type",
      "description": "The specific probate form being completed",
      "type": "categorical",
      "values": [
        "P-100 Petition for Probate",
        "P-110 Notice of Hearing",
        "P-120 Letters Testamentary",
        "P-130 Inventory and Appraisal",
        "P-140 Final Account and Petition for Distribution"
      ],
      "required": true,
      "authority_source": "Alaska Court System Form Catalog"
    },
    {
      "name": "decedent_status",
      "description": "Whether the deceased left a valid will",
      "type": "categorical",
      "values": ["testate", "intestate"],
      "required": true,
      "authority_source": "Alaska Stat. § 13.16.005"
    },
    {
      "name": "asset_type",
      "description": "Categories of property in the estate",
      "type": "array",
      "values": [
        "real_property",
        "bank_accounts",
        "vehicles",
        "personal_property",
        "stocks_bonds",
        "retirement_accounts"
      ],
      "required": true,
      "authority_source": "Alaska Stat. § 13.16.095"
    },
    {
      "name": "institutional_reference",
      "description": "Specific organization mentioned in guidance (law school, court, agency)",
      "type": "string",
      "required": false,
      "validation": "Must exist in verified institutional registry"
    }
  ]
}

Required State

{
  "required_state": [
    "jurisdiction",
    "form_type",
    "decedent_status",
    "asset_type"
  ],
  "validation": {
    "jurisdiction": "Must match user's filed petition jurisdiction",
    "form_type": "Must exist in current Alaska Court System form catalog",
    "institutional_existence": "Any mentioned organization must have verified existence"
  }
}

Authority Requirements

{
  "authority_requirements": {
    "jurisdictional_rules": {
      "source": "Alaska Probate Code (AS 13.06-13.36)",
      "update_frequency": "Legislative session",
      "verification": "Cross-reference with Alaska State Legislature database"
    },
    "procedural_forms": {
      "source": "Alaska Court System TF-920 Probate Forms Manual",
      "update_frequency": "Annual or as amended",
      "verification": "Compare form numbers and titles against official court catalog"
    },
    "institutional_registry": {
      "source": "Alaska Commission on Postsecondary Education (ACPE) institutional list",
      "update_frequency": "Continuous",
      "verification": "Verify existence before mentioning any educational institution, court office, or state agency"
    },
    "fee_schedules": {
      "source": "Alaska Administrative Rule 12 AAC 56",
      "update_frequency": "Annual",
      "verification": "Current filing fees for specific form types"
    }
  }
}

Completeness Gates

1. Institutional Existence Verification

What Alaska AVA Was Missing

{
  "gate_id": "institutional_existence_check",
  "trigger": "System generates response mentioning specific organization",
  "required_state": ["institutional_reference"],
  "authority_check": {
    "source": "Alaska institutional registries (ACPE for schools, Alaska Bar for attorneys, Court System for courts)",
    "validation": "Organization name must match verified registry entry",
    "failure_mode": "Block response if institution cannot be verified"
  },
  "alaska_ava_failure": {
    "what_happened": "Hallucinated 'Alaska School of Law' (does not exist)",
    "why_it_failed": "No gate to verify institutional existence before mentioning it",
    "correct_behavior": "System should have checked ACPE registry, found no match, and either (1) omitted law school reference or (2) returned error requiring human review"
  }
}

2. Procedural Accuracy Gate

Form-to-Situation Validation

{
  "gate_id": "form_situation_match",
  "trigger": "System recommends specific probate form",
  "required_state": [
    "jurisdiction",
    "form_type",
    "decedent_status",
    "asset_type"
  ],
  "authority_check": {
    "source": "Alaska Court System Form Selection Matrix (TF-920)",
    "validation": "Recommended form must be appropriate for user's situation (testate vs intestate, asset types)",
    "failure_mode": "Prevent form recommendation if situation parameters don't match form requirements"
  },
  "example": {
    "user_state": {
      "jurisdiction": "Alaska",
      "decedent_status": "intestate",
      "asset_type": ["bank_accounts"]
    },
    "invalid_output": "File P-100 with attached will (user has no will)",
    "valid_output": "File P-105 Petition for Letters of Administration (intestate estate < $100K)"
  }
}

3. Jurisdictional Constraint Gate

State-Specific Procedural Requirements

{
  "gate_id": "jurisdictional_procedures",
  "trigger": "System provides guidance on probate process",
  "required_state": ["jurisdiction"],
  "authority_check": {
    "source": "Alaska Stat. Title 13 (Decedents' Estates, Guardianships, Transfers, Trusts)",
    "validation": "Procedural steps must match Alaska-specific requirements, not general probate principles",
    "failure_mode": "Reject output that uses procedures from other states or generic 'probate best practices'"
  },
  "cross_state_conflicts": {
    "Alaska": "Requires 30-day notice period (AS 13.16.460)",
    "California": "Requires 120-day notice period (CA Prob Code § 8110)",
    "error_prevention": "Gate ensures Alaska-specific timeline is provided when jurisdiction=Alaska"
  }
}

4. Current Law Verification Gate

Statute/Rule Version Control

{
  "gate_id": "current_law_check",
  "trigger": "System cites specific statute or court rule",
  "required_state": ["jurisdiction"],
  "authority_check": {
    "source": "Alaska State Legislature current statute database",
    "validation": "Cited statute number must exist in current version, not repealed or amended",
    "failure_mode": "Flag citations that don't match current law for SME review"
  },
  "drift_detection": {
    "monitoring": "Track when cited statutes were last verified against official source",
    "update_trigger": "Re-verify all statute references after each legislative session",
    "failure_example": "Citing repealed AS 13.11.010 (superseded in 2017 revision)"
  }
}

Output Structure

{
  "system_output": {
    "recommended_form": "P-100",
    "filing_location": "Alaska Superior Court, [User's Judicial District]",
    "required_attachments": [
      "Death certificate",
      "Will (if testate)",
      "List of heirs"
    ],
    "procedural_steps": [
      "1. File P-100 with court clerk ($200 filing fee per 12 AAC 56.030)",
      "2. Publish notice in local newspaper (AS 13.16.460 - 30 day period)",
      "3. Serve notice to all heirs by certified mail",
      "4. Attend hearing (date set by court)"
    ],
    "grounding": {
      "form_catalog": "Alaska Court System TF-920 (2025 edition)",
      "fee_source": "12 AAC 56.030(a)(1)",
      "notice_requirement": "AS 13.16.460",
      "service_requirement": "Alaska Prob Rule 4"
    },
    "completeness_verification": {
      "institutional_checks_passed": true,
      "jurisdiction_validated": "Alaska",
      "form_exists_in_catalog": true,
      "all_required_state_provided": true
    }
  }
}

Implementation Guidance

For Legal Self-Help Developers

  1. Never Mention Unverified Institutions

    • Maintain registry of verified courts, law schools, bar associations
    • Cross-check any organizational reference before inclusion in response
    • If institution cannot be verified, omit reference or route to human reviewer

  2. Validate Form Recommendations

    • Build decision tree mapping user situations to appropriate forms
    • Ground in official court form selection matrices
    • Prevent form suggestion if required state is missing

  3. Enforce Jurisdictional Boundaries

    • Tag all procedural guidance with specific jurisdiction
    • Block cross-jurisdiction contamination (California rules leaking into Alaska advice)
    • Maintain separate knowledge bases per state

  4. Track Authority Source Freshness

    • Log when each statute/rule was last verified
    • Trigger re-verification after legislative sessions
    • Surface staleness warnings before generating guidance

Monitoring & Drift Detection

{
  "drift_indicators": [
    {
      "metric": "form_catalog_staleness",
      "threshold": "90 days since last verification",
      "action": "Re-scrape Alaska Court System form catalog"
    },
    {
      "metric": "statute_citation_failures",
      "threshold": "> 5% of citations fail current law check",
      "action": "Full statute database refresh"
    },
    {
      "metric": "institutional_mentions",
      "threshold": "Any organization mentioned without registry verification",
      "action": "Immediate response blocking + alert"
    }
  ]
}

Lessons from Alaska AVA

What Went Wrong

  • No Institutional Verification Gate: System could mention organizations without existence check
  • Retrieval ≠ Grounding: RAG provided relevant context but no structural validation
  • Scope Creep Without Gates: Expanding from narrow forms to general advice increased hallucination risk

What Would Have Prevented It

  • Completeness gate requiring institutional references to pass registry lookup
  • Form-to-situation validation preventing incorrect form recommendations
  • Jurisdictional constraint enforcement blocking cross-state procedure leakage
  • Current law verification catching outdated or nonexistent statute citations

Broader Implications

Legal self-help is a test case for regulated advice domains:

  • Tax preparation (IRS form validation)
  • Immigration applications (USCIS procedure grounding)
  • Benefits eligibility (state-specific rule enforcement)
  • Healthcare directives (jurisdictional living will requirements)

All share the same failure mode: approximation kills. You cannot "mostly" get a statute number right. Gates enforce exact matches or hard stops.

Reference: SAF Incident #021 - Alaska Virtual Assistant (AVA) Probate Chatbot
Authority Sources: Alaska Statutes Title 13, Alaska Court System TF-920, ACPE Institutional Directory
Last Updated: January 2026