AI governance roles: who is responsible for what?

By Ivo Donker — compiled with AI assistance (Claude & Gemini)

Many artificial intelligence initiatives don't fail because of technical complexity or missing algorithms, but because of an organizational vacuum. When a model generates unexpected outcomes, processes privacy-sensitive data, or provides customers with faulty advice, the question of who is actually liable and responsible arises immediately. Without clearly assigned roles, a dangerous dynamic emerges within organizations: IT points to the business for the use case, the business points to data science for model quality, and compliance only flags risks once a system is already live.

Setting up AI governance is about assigning concrete decision-making authority and operational responsibilities across the entire lifecycle of an algorithm. This requires a balance between the speed of innovation and risk management. In this article, we cover the essential roles within a mature AI governance structure, how these roles come together in a RACI matrix, and how mid-sized organizations can allocate these tasks without setting up a heavy bureaucratic layer.

The three governance layers of AI governance

To distribute responsibilities effectively, it is necessary to distinguish between three governance layers: strategic, tactical, and operational. Each level has a different level of abstraction, different objectives, and its own frequency of oversight.

At the strategic level , the executive board or leadership determines the frameworks within which AI may be deployed. Here, risk appetite, ethical boundaries, and strategic investment priorities are established. This level determines which domains are categorically excluded from automated decision-making and approves the overarching policy. Those looking to start formalizing these frameworks can draft an AI policy for the organization to immediately establish clear principles and ethical guidelines for all employees.

The tactical level translates these strategic principles into actionable standards, validation procedures, and approval criteria. Here, a multidisciplinary panel assesses whether proposed use cases fall within the established risk tolerance, whether the European AI Act is being complied with, and whether the necessary data protection techniques have been applied. This body acts as a gatekeeper for new AI projects.

The operational level covers the day-to-day build, integration, monitoring, and use of the systems. This is where engineers, product owners, domain experts, and end users operate. They are responsible for applying the guidelines in code, monitoring data quality, validating model performance, and flagging anomalies during production.

The AI Ethics & Governance Board: mandate and composition

The AI Ethics & Governance Board (also known as the AI steering committee) forms the central decision-making body at the tactical level. Without a clear mandate, such a committee quickly becomes a toothless advisory board where projects get delayed without risks actually being mitigated.

The board's mandate must be explicitly documented and include at least the following powers:

• Approval authority (gatekeeping): No AI system may go into production without a risk assessment formally approved by the board.
• Stop mandate (kill switch): The authority to immediately shut down an operational system if serious drift, bias, security incidents, or policy violations are identified.
• Exception management: Assessing and, where appropriate, temporarily permitting deviations from standard compliance guidelines.

The effectiveness of the board hinges on its composition. A one-sided composition creates blind spots: a purely technical board underestimates legal and reputational risks, while a board made up solely of lawyers misses technical feasibility and operational context. A mature composition includes representatives from Legal/Compliance, IT Security (CISO), Data Science/Engineering, the leading business unit, and HR or an ethics reviewer.

The AI Solution Owner: business and functional ownership

Every AI system needs a single, undivided owner: the AI Solution Owner (often a specialized Product Owner or domain manager). This person is fully responsible for the business value, performance, and responsible use of the specific system throughout its entire lifecycle.

The AI Solution Owner doesn't necessarily need to be a technical expert, but must thoroughly understand the model's workings, assumptions, and limitations. Core tasks include:

• Defining the system's objectives, acceptance criteria, and minimum accuracy thresholds.
• Ensuring the system is used exclusively for the purpose it was designed for (preventing scope creep and function creep).
• Allocating budget and capacity for ongoing monitoring, evaluation, and recalibration of the model.
• Escalating incidents to the Governance Board in a timely manner and formulating an exit strategy.

A common pitfall is that ownership evaporates after go-live. When a project team is disbanded, a model often keeps running without active oversight. To prevent performance degradation from going unnoticed, the resource on post-go-live management and ownership of AI in production describes how this responsibility is organizationally safeguarded.

The Privacy Officer and Legal Counsel: legislation and compliance

The legal and privacy-related aspects of AI require active involvement from the Data Protection Officer (DPO), Privacy Officer, and Legal Counsel. Their role is shifting from a traditional, one-off GDPR review to continuous oversight of data flows, intellectual property, and compliance with the European AI Act.

Core tasks within this domain include:

• Risk classification under the AI Act: Determining whether a system falls into the minimal, specific transparency risk, high-risk, or unacceptable risk category.
• Data Protection Impact Assessments (DPIA): Assessing the lawfulness of data processing, particularly when training, fine-tuning, or searching personal data via RAG.
• Rights of data subjects: Ensuring that data subjects can exercise their rights, including the right to human intervention and an explanation in automated decision-making (Article 22 GDPR).
• Vendor contracts and intellectual property: Reviewing data processing agreements, license terms, and intellectual property rights on both training data and model outputs.

The Chief Information Security Officer (CISO) and IT security

Machine learning and LLM architectures introduce new attack vectors that fall outside the traditional IT security domain. The CISO and security engineers bear responsibility for securing the AI pipeline against manipulation, data breaches, and unauthorized access.

Within AI governance, security specifically focuses on:

• Adversarial robustness: Protection against prompt injection (direct and indirect), model inversion attacks, and data poisoning during data ingestion.
• Access management and data segregation: Guaranteeing that an AI model or retrieval component (such as a vector database) respects the existing authorization structure and doesn't leak data across departmental boundaries.
• API and key management: Securing endpoints, rate limiting, and secure storage of authentication tokens for external and internal model providers.

The Data & Model Steward: quality, drift, and documentation

Where the CISO and Legal Counsel oversee frameworks, the Data & Model Steward focuses on the technical and substantive integrity of the underlying data and models. This role forms the bridge between data science and governance.

The tasks of the Data & Model Steward include:

• Data quality and lineage: Documenting the origin, processing steps, and quality of training and validation datasets.
• Drift monitoring: Periodically measuring concept drift and data drift to determine whether reality has changed relative to the training data.
• Bias and fairness audits: Conducting statistical tests to verify that the model doesn't exhibit systematic, unlawful discrimination against specific subgroups.
• Model cards and documentation: Maintaining technical model cards that uniformly document architecture, performance benchmarks, hyperparameters, and known limitations.

The Domain Expert and End User: human oversight

Without trained end users and subject-matter domain experts, "human-in-the-loop" is nothing more than a paper tiger. When employees blindly adopt a model's output (automation bias) or, conversely, structurally distrust and bypass the system, governance fails on the shop floor.

Domain experts and operational users are responsible for:

• Validating the substantive content of generated results before final processing or sending to customers.
• Recording hallucinations, incorrect classifications, or quality deviations through structured feedback channels.
• Flagging changes in work processes that may affect the model's relevance.

To fulfill this role effectively, employees need sufficient AI literacy and clear instructions. The overview on AI adoption in teams and change management explains in more detail how teams are trained to work effectively and critically with AI tools.

Autonomous systems and agentic workflows: a shift in responsibility

With the rise of autonomous AI systems that independently plan tasks, call tools, and execute decisions, the nature of governance is changing fundamentally. Where traditional models delivered a static prediction, autonomous agents take successive actions in external systems such as CRM and ERP databases.

This shift toward agentic AI and autonomous systems forces organizations to set stricter frameworks around mandate and freedom of action. In autonomous workflows, a human cannot check every individual action in advance. Governance therefore shifts from the task level to the system level:

• Financial and operational limits: Setting hard thresholds for actions an agent may perform without human approval (for example, transaction limits or permissions to modify data).
• Audit trails of decision chains: Complete logging of intermediate steps, reasoning, and tool calls to be able to reconstruct afterward why an agent arrived at a specific action.
• Failsafe mechanisms: Automatically pausing processes as soon as an agent steps outside its confidence interval or shows unusual patterns.

In this landscape, the role of the Agent Supervisor becomes crucial: a specific officer who monitors the behavior of a fleet of active software agents and is authorized to manually adjust or interrupt processes.

The RACI matrix for AI systems

To eliminate ambiguity between departments, the RACI matrix below maps out the key governance tasks across the lifecycle of an AI application. Here, R stands for Responsible, A for Accountable, C for Consulted, and I for Informed.

In this matrix, it stands out that the AI Solution Owner is the Accountable (A) party for nearly every project phase. After all, only one person can be ultimately accountable for the business success and compliance of a specific system. The other roles provide specialized expertise (R or C) to help fulfill that responsibility.

Implementation in SMEs: combining roles without conflicts of interest

In mid-sized organizations, it's often impossible to appoint a full-time officer for every sub-task. In practice, roles are combined. This is organizationally feasible, provided fundamental conflicts of interest are taken into account.

Safe combinations that work well in practice:

• The Privacy Officer also fulfills the role of Compliance Lead for the AI Act.
• The Lead Data Engineer takes on the tasks of the Data & Model Steward as well.
• The Product Manager of a domain acts as AI Solution Owner for the AI features within that product.

Combinations that must absolutely be avoided due to conflicts of interest:

• Combining Solution Owner with Compliance/Privacy: Whoever is responsible for commercial delivery and deadlines cannot independently judge privacy or ethical risks.
• Combining Developer with Model Validator: The person who trains or integrates the model must not be the only party that performs and approves the quality and bias tests.

Step-by-step plan for formalizing roles

To operationally embed AI governance in the organization, follow these four consecutive steps:

1. Inventory of active systems: Map out all AI models, tooling, and API integrations, and immediately assign one explicit Solution Owner to each system.
2. Establishment of the approval panel: Form the AI Governance Board with permanent representatives from business, IT, security, and legal, and formally document its decision-making mandate.
3. Documenting escalation paths: Determine who is alerted in case of deviations, who is authorized to shut down a model, and within what response time an incident must be resolved.
4. Periodic audit and recalibration: Evaluate every six months whether the division of roles still aligns with the latest legislation, technological standards, and the growth of the AI portfolio.

A well-thought-out governance structure doesn't slow down innovation — it actually accelerates it, because teams know exactly within which boundaries they can experiment autonomously and when formal review is required.