AI Security Compliance and Risk Assessment Framework for Large Language Model Systems.

Abstract

The rapid integration of Large Language Models into organizational workflows has outpaced the governance frameworks designed to oversee them. Standards such as NIST AI RMF 1.0 and ISO/IEC 42001 provide high-level principles for responsible AI but lack the operationalized, testable controls required for consistent compliance auditing of deployed LLM systems. This thesis designs, implements, and empirically evaluates an AI-powered compliance assessment framework that operationalizes these standards into a structured, interactive audit system.The framework is grounded in a Design Science Research methodology. Its core artifact is an AI audit agent backed by a 30-control library spanning six risk domains, a 0-to-5 maturity scoring model anchored to the NIST SP 800-53A evidence hierarchy, and a rule-based adaptive selection layer that tailors the control set to the deployment context without sacrificing reproducibility. A key research finding is that NIST AI RMF 1.0 contains no dedicated subcategories for three OWASP LLM Top 10 (2025) risk categories: System Prompt Leakage (LLM07), Vector and Embedding Weaknesses (LLM08), and Unbounded Consumption (LLM10). This thesis constructs three composite controls to close this gap, a finding independently corroborated by NIST AI 600-1 (July 2024). The agent is evaluated across three synthetic deployment scenarios representing low, moderate, and high organizational maturity, producing compliance scores of 23 percent, 33 percent, and 71 percent respectively. Two independent human assessors validated all 90 control assessments, producing an inter-assessor agreement rate of 72.2 percent and an LLM-to-human consensus agreement rate of 81.1 percent, confirming calibration within the expected range for AI-assisted assessment instruments. This thesis contributes to the field in four respects. First, it provides the first operationalized, evidence-requesting 30-control compliance library that achieves full coverage of the OWASP LLM Top 10 (2025) with explicit standard citations to NIST AI RMF, ISO/IEC 42001, and ISO/IEC 27001. Second, it identifies and formally documents a structural gap in NIST AI RMF 1.0 through three composite controls whose necessity is corroborated by NIST AI 600-1. Third, it demonstrates a replicable adaptive selection architecture that separates deterministic compliance logic from LLM-assisted language generation. Fourth, it establishes a validated empirical baseline characterizing AI compliance tool calibration behavior across the full maturity range, with two specific calibration gaps confirmed by independent human assessors for future scoring model refinement.