Certified: The ISACA AAIA Audio Course

In this episode, we take on a topic that sounds like pure administration at first, but it quietly determines whether an A I system can be trusted at all once it is running in the real world. Configuration Management (C M) is the discipline of knowing exactly what you have, how it is set, who can change it, and how you can prove what changed when outcomes shift. That matters in A I because the model is only one ingredient in the system, and the behavior you see in production comes from a combination of code, data, model artifacts, and all the settings that connect them. A beginner can think of it like baking, because even if you use the same recipe, a small change in oven temperature, ingredient measurements, or timing can completely change the result, and the person eating the cake only cares about the outcome. When organizations struggle with A I incidents, they often discover they cannot answer basic questions like which model version was used, what data transformations were applied, or which threshold setting decided a borderline case. Evaluating C M across code, data, and models is therefore about proving the organization has control over the moving parts that shape decisions, not merely control over a single file called the model.

Before we continue, a quick note: this audio course is a companion to our course companion books. The first book is about the exam and provides detailed information on how to pass it best. The second book is a Kindle-only eBook that contains 1,000 flashcards that can be used on your mobile device or Kindle. Check them both out at Cyber Author dot me, in the Bare Metal Study Guides Series.

A useful starting point is to understand what configuration really means in an A I system, because beginners sometimes imagine it as a small list of settings in a menu. In practice, configuration includes any variable choice that changes behavior without rewriting the whole system, such as feature selection rules, data cleaning thresholds, model parameters, routing logic, and decision cutoffs. It also includes environment settings that shape execution, like which data source is read, which model registry is used, and what fallback behavior happens when inputs are missing. The reason C M is so central is that A I behavior can shift even when the code looks unchanged, simply because a configuration file points to a different dataset or because a threshold value is tuned to meet a performance target. An evaluator wants to know whether the organization treats these settings as controlled assets or as casual knobs that anyone can turn when a metric looks off. When configuration is uncontrolled, accountability disappears, because you cannot connect outcomes to intentional decisions. Strong C M restores accountability by making configuration visible, versioned, reviewed, and reproducible.

Because Task 14 is about evaluation, it helps to frame the evaluator’s job as three questions that are easy to say and hard to answer without discipline. The first question is whether the organization can inventory and identify all the configuration that influences an A I decision, including code settings, data settings, and model settings. The second question is whether the organization can prove what configuration was in effect for a particular outcome, meaning it can trace a decision back to a specific model version, data snapshot, and configuration set. The third question is whether changes to that configuration are controlled, meaning changes are authorized, tested, documented, and reversible. Beginners often think evaluation means reading a policy document, but with C M, the evaluation must focus on evidence that configuration control exists in practice. Evidence shows up in how the organization stores configuration, how it reviews changes, how it restricts access, and how it reconstructs past states. Without that evidence, the system is a black box that can change quietly, which is exactly what auditors and assurance professionals are trained to treat as unacceptable risk.

To evaluate configuration management across code, you first need to understand how code configuration differs from code logic. Code logic is the actual implementation, while code configuration is the set of choices that tell the code how to behave in a given environment, such as which model endpoint to call or which thresholds to use for escalating decisions. Organizations commonly store these choices in configuration files, environment variables, or external services, and that can be a strength when it is controlled and a weakness when it is not. A mature organization treats these settings as part of the release artifact, meaning they are versioned and tied to a particular release of the system. An evaluator will look for whether configuration changes follow the same review discipline as code changes, because unreviewed configuration changes can cause the same harm as unreviewed code. Another key point is whether the organization can detect unauthorized or accidental configuration drift, where a setting changes without a formal change record. For beginners, the big lesson is that code can be stable while behavior changes, and code configuration is one of the most common reasons why.

When you shift to data configuration, the evaluation becomes even more important, because data is the fuel that powers A I behavior. Data configuration includes which data sources are used, what time windows are included, how missing values are handled, how categories are mapped, and what transformations turn raw information into features the model consumes. A small change here can have a huge impact, such as dropping a field that the model relies on or changing a mapping that flips categories. Beginners might assume data is a fixed thing, but in real systems data pipelines evolve, vendors change formats, and business processes introduce new values that were not present before. An evaluator therefore checks whether the organization has controlled definitions for features and transformations, and whether those definitions are versioned so the organization can reproduce what the model saw at a given time. This is also where monitoring intersects with C M, because monitoring might reveal drift, but C M tells you whether the drift came from the world changing or from the organization changing its own data handling. Strong data configuration control is what prevents silent changes from turning into silent bias or silent failure.

Model configuration is the third pillar, and it includes the model version, its parameters, its calibration settings, and the operational choices around it, such as confidence thresholds, ensemble weighting, or fallback behavior when confidence is low. Even if a beginner never trains a model, they can still understand that models are not just files, they are bundles of learned patterns plus settings that shape how those patterns are used. An evaluator looks for whether model artifacts are stored in a controlled registry, whether each model is uniquely identified, and whether the organization can prove which model was active at a given time. They also look for whether model configuration includes the context needed to interpret outputs, such as the intended use case, known limitations, and performance boundaries. A major risk is the model that gets updated or swapped without a clear record, because then changes in outcomes can be denied or misunderstood. Another risk is the model that is correct in a narrow sense but is used with an inappropriate threshold setting, causing an unacceptable error tradeoff. Evaluating model configuration management means verifying that model choices are visible, traceable, and governed, not treated as experimental tweaks.

One of the most powerful ways to explain configuration management to a beginner is to connect it to reproducibility, which is the ability to recreate the system’s behavior later in order to investigate, validate, or audit it. Reproducibility depends on knowing exactly which code version was used, which data snapshot or dataset version was used, and which model artifact and settings were active. If any one of those is missing, the organization cannot reliably replay a decision, and that makes incident investigation and accountability much harder. Evaluators look for whether the organization captures the right metadata, such as model identifiers, dataset identifiers, and configuration hashes, because these are the anchors that tie decisions to a specific system state. They also look for whether the organization can actually perform a reproduction exercise, because a claim of reproducibility is weaker than a demonstrated reproduction. For beginners, a helpful analogy is keeping notes in a science experiment, because without notes you cannot prove your results or learn from mistakes. In A I assurance, reproducibility is how the organization proves that outcomes come from controlled choices rather than from chaos.

Another central evaluation point is change control, meaning how configuration is allowed to change and how the organization prevents untracked change. In well-governed environments, configuration changes require requests, review, approval, and testing that matches risk, because configuration changes can alter who gets impacted and how. An evaluator checks whether configuration changes are reviewed by the right roles, especially when they affect decision thresholds, sensitive features, or policies tied to fairness and safety. They also examine whether the organization uses separation of duties, meaning the person who proposes a configuration change is not always the person who approves and deploys it. For beginners, separation of duties is like having one person count cash and another person verify the count, because it reduces error and discourages misuse. A strong control environment also tracks exceptions, because emergencies happen, but emergency configuration changes must be recorded and reviewed afterward. The overall goal is that configuration changes are intentional, visible, and accountable, even when the organization is moving quickly.

Access control is another major part of configuration management evaluation, because configuration is power. If someone can change a threshold, redirect a data source, or swap a model artifact, they can change outcomes in ways that affect real people and business decisions. Evaluators therefore check who can read and write configuration, how privileges are granted, and whether access is limited to what people need to do their jobs. They also look for strong authentication and logging, because if configuration is changed improperly, the organization must be able to trace who changed what and when. Beginners should understand that access is not only about preventing malicious insiders, but also about preventing accidents, because mistakes happen when too many people have broad rights. Another subtle issue is the use of shared accounts or informal credential sharing, which destroys accountability and makes audits nearly impossible. A mature environment can show clean access records, change logs, and evidence that configuration repositories are protected like critical assets. If configuration access is loose, then every other control is weaker because the system can be altered quietly.

Evaluation also needs to consider the boundaries between environments, because what is safe in testing is not automatically safe in production. Many organizations have separate environments for development and production, and configuration should reflect that separation, with strict rules about what can move across boundaries. An evaluator will look for controls that prevent test configurations, test models, or test data sources from being used in production by mistake. They also look for controls that ensure production configuration changes are not made casually or directly, because direct edits increase the risk of misconfiguration and reduce traceability. For beginners, this is similar to the difference between practicing in a rehearsal room and performing on stage; the environment changes what is acceptable, and the process must respect that. Another important point is that configuration should be consistent where it must be consistent, so that the organization can compare outcomes meaningfully over time. If production configuration is unstable, it becomes difficult to interpret monitoring signals because the baseline keeps moving. Strong C M keeps environment boundaries clear and keeps production configuration stable except through controlled changes.

A beginner-friendly misunderstanding to address is the idea that configuration management is mostly about documentation. Documentation matters, but C M is primarily about control and evidence, which means documentation must be supported by systems and processes that enforce it. If a document says configuration changes require approval but people can change production settings directly without review, the document is not a control. Evaluators therefore look for objective evidence, such as recorded change requests, version history, and automated checks that block unapproved changes. They also look for whether documentation is usable, meaning it clearly explains what each configuration setting does and what risks are associated with changing it. Good documentation reduces the chance of someone making a change without understanding its impact, which is especially important in A I because the impact might be indirect or delayed. For beginners, it is enough to remember that a policy is a promise, but evidence is the proof. Configuration management succeeds when the proof matches the promise.

Another important concept is configuration drift, which is when the system’s actual settings slowly diverge from what the organization believes they are. Drift can happen through manual hotfixes, emergency changes that were never reconciled, or gradual accumulation of small tweaks made to chase performance metrics. Evaluators check whether the organization detects drift through periodic reconciliation, automated validation, or continuous monitoring of configuration state. They also examine whether the organization has clear baselines for expected configuration and clear procedures for restoring baseline if drift is detected. This matters because drift undermines predictability, and predictability is essential for governance, fairness assessment, and incident response. Beginners can think of drift like gradually changing household rules without telling everyone, until no one agrees on what the rules are anymore. In A I, drift can mean a model that seems to be behaving differently, and without C M, the organization cannot quickly tell whether that difference is due to data changes, model changes, or configuration changes. Evaluating drift control is therefore part of evaluating whether the organization can keep an A I system stable and trustworthy.

To make this practical, imagine an A I system that helps prioritize which cases should be reviewed by a human team. The organization might change a threshold setting to reduce workload, change a data transformation to handle a new input format, and deploy a model update to improve detection, all within a few weeks. If those changes are not versioned and tied together, the organization may see outcomes shift and be unable to explain why, leading to confusion, blame, and delayed correction. An evaluator would ask whether each change was captured as part of configuration management, whether the combined change was tested as a system, and whether the organization can trace a particular decision back to the exact configuration state. They would also ask whether access controls prevented unauthorized tweaks and whether monitoring and incident triggers were updated to reflect the new configuration. This example shows how code, data, and model configuration interact, because the real behavior emerges from the combination, not from any single element. For beginners, the takeaway is that configuration management is how an organization keeps its A I system understandable, even as it evolves.

When you step back, evaluating configuration management for A I across code, data, and models is about proving the organization has practical control over the elements that shape outcomes. The evaluator looks for inventory and traceability, so the organization can identify what is running and reproduce decisions later. They look for controlled change processes, so configuration updates are reviewed, tested, approved, and reversible. They examine access controls and logging, because configuration is a high-impact asset that must be protected from mistakes and misuse. They verify environment separation and drift detection, because production stability depends on disciplined boundaries and consistent baselines. For brand-new learners, the central lesson is that A I risk is not only about the model’s intelligence, but about the organization’s ability to manage the system’s settings and dependencies responsibly over time. If you can explain how C M connects code, data, and models into a traceable, controllable system, you are building the core Task 14 mindset: trustworthy A I requires not just good models, but rigorous control over the configuration that makes those models act in the real world.