The notes of Justin Abrahms

Recently updated

Axes of LLM Difficulty

2 min read

via https://youtu.be/8jKAT8GNDE0

  1. Reasoning Problems (The “Intelligence” Axis)
    • Pure/Novel Reasoning: Solving logic problems never encountered before (e.g., ARC-AGI2 benchmark).
    • Scientific Discovery:
      • Drug Discovery: Predicting protein structures and drug-binding sites (e.g., AlphaFold).
      • Mathematics: Disproving long-standing conjectures through combinatorial counter-examples.
      • Physics: Calculating gravitational radiation or optimizing crystal growth.
    • Specialized Quantitative Business Tasks:
      • Multi-jurisdiction Tax Optimization: Navigating interacting global tax codes.
      • Complex Derivative Pricing: Advanced financial mathematics.
      • Structural Fraud Detection: Tracing complex chains of transactions across multiple entities.
      • Regulatory Compliance: Determining if new instruments trigger specific, overlapping legal obligations.
  2. Effort & Scaling Problems (The “Endurance” Axis)
    • Large-Scale Data Audits: Reviewing thousands of vendor contracts for specific compliance changes.
    • Legacy Migrations: Moving millions of lines of code (e.g., COBOL) to modern frameworks.
    • Massive Surface Area Analysis: Reviewing every customer interaction from a quarter to find churn signals.
    • Autonomous Execution: Sustained work over days or weeks without human intervention (e.g., closing engineering tickets).
  3. Coordination & Logic-Action Problems (The “Orchestration” Axis)
    • Agent Management: Coordinating teams of AI agents to build complex systems (e.g., a C compiler).
    • Organizational Awareness:
      • Routing work across dependencies (Back-end vs. Front-end vs. QA).
      • Understanding which human team “owns” a specific repository or context.
    • Tool Orchestration: Combining raw intelligence with web search, code execution, and database access to complete real-world office tasks.
  4. Experience-Based Problems (The “Domain Expertise” Axis)
    • Pattern Recognition: Debugging faster not through logic, but because of “having seen that stack trace before.”
    • Internalized Context: Knowing which legal “boilerplate” clauses are actually litigated versus those that are never enforced.
    • Historical Intuition: Remembering specific production incidents or past deal structures that inform current strategy.
  5. Ambiguity & Judgment Problems (The “Strategy” Axis)
    • Problem Definition: Figuring out what the customer actually needs when they cannot articulate it themselves.
    • Strategic Intuition: Holding multiple incomplete mental models in tension until a direction resolves.
    • Product Sense: Deciding what to build when market signals are contradictory.
  6. Human-Centric Problems (The “Soft Skills” Axis)
    • Emotional Intelligence (EQ):
      • Delivering difficult feedback to employees facing personal crises.
      • Reading “boardroom silence” to detect unspoken opposition.
      • Managing team anxiety during a reorganization.
    • Willpower & Courage:
      • Making the “politically dangerous” but strategically correct call.
      • Choosing to kill a project after months of investment because the market shifted.
    • Validation & Authority: The expert judgment required to verify if an AI’s “novel” reasoning is actually correct and safe to act upon.

Graph View