Like the NZISM, the NZAIM should delineate the different types of algorithm and AI techniques

Methodical set of instructions that can be executed by a machine. Algorithms are typically authored by humans but increasingly can be written by generative AI.

Subsets (non-exhaustive): goal-driven optimisation, artificial intelligence

Examples: any logic that runs on machines, from a simple business rule to categorise an email, to a generative AI model powering advanced features in existing software.

Evaluation: varies by technique. At a high level, algorithms are validated using traditional software testing frameworks. Test frameworks typically distinguish between verifying the outputs of the system and validating the outcomes the system sought to achieve. Tests are typically manually written but increasingly can be written by advanced AI.

Challenges:

• wide definition makes it hard to track for governance purposes
• lack of continuous monitoring risks performance degradation as the operating environment changes from when the rules were developed (concept drift) or encounters new data it cannot properly handle (data drift)
• lack of documentation may deny individuals their official information rights to rules that lead to, or the reasons for, decisions or recommendations
• poor auditability without an audit trail required for all public records, including those generated by an algorithm
• automation bias may result in uncritical human deference to automatic predictions
• appeals, judicial review and Ombudsman investigations may be taken against a decision, recommendation or action performed by an algorithm, ensuring that such acts were made lawfully. A degree of reversibility may be required.

Algorithms consisting of predefined logic, rules or ‘symbols’ (as in the traditional category of symbolic AI) for reaching an output. These algorithms are typically manually designed by humans but may be designed by large language models through linguistic likelihood, rather than mathematical derivation. This category excludes data-derived systems that may itself consist of rules or symbols, such as decision tree classifiers.

Examples: expert systems, business rules, predefined decision trees

Evaluation: as for algorithms

Challenges: relative simplicity and explainability may come at the expense of accuracy.

Algorithms that search for the best solution to a defined problem, which becomes exponentially difficult when attempted exhaustively. They typically use simulations to model a solution’s interaction with its environment. The “best solution” may be the end intervention itself, such as a set of school bus routes that maximise coverage and minimise resource use. It may also be the best strategy for how actors behave and react to an intervention.

Actors⁵ optimise their behaviour based on their own motivation (e.g. economic gain, travel time minimisation, personal health) reaching equilibrium against others in the simulation.

Subsets: reinforcement learning, evolutionary computation.

Examples: navigation application planning, supply chain optimisation, school bus route generation (Ministry of Education’s School Route Transport Optimiser), transport system modelling (Ministry of Transport’s Monty), large population models (PHF Science’s ALMA)

Evaluation: the output is necessarily the ‘best’ given the problem and search constraints, but further validation is required to ensure the desired outcome was met (cf. impact evaluation in policy design) [Challenges continues next page]

Challenges:

• defining the true problem, constraints and objectives are difficult, requires translating end-user requirements to measurable equations, which can often be incomplete, framed poorly or from a deficit basis, can change over time
• representativeness of simulated population and behaviour to actual population
• computationally expensive, often using AI methods to more quickly find a sufficiently (not always the most) effective solution

Machine-based systems that infer (based on implicit or explicit objectives) from the input it receives how to generate outputs (predictions, content, recommendations, decisions, actions).

Subsets: machine learning, evolutionary computation

Examples: from basic linear regressions or probability models that predict numbers or outcomes; to advanced deep neural networks that can generate complex nuanced text and images, or guide decision-making of simulated actors.

Evaluation / challenges: varies by technique

AI models that are automatically developed using existing observations, understanding and approximating how that data contributes to an output.

Subsets: supervised learning, unsupervised learning, reinforcement learning, deep learning, generative AI

Evaluation: self-evaluates its output at each iteration of model development using the objective it was given, but further manual evaluation is typically required to assess whether the objective has produced the right outcome (cf. impact evaluation in policy design)

Challenges:

• data accuracy – predictions are only as reliable as the underlying data, risk of systemic error (e.g. police non-completion of callout assessment correlated with victim ethnicity), human error (e.g. staff assumes data points instead of asking the subject), and faulty assumptions (e.g. using outputs as proxy for outcomes).
• data robustness – data must be representative of the population and operating environment which a model will be applied (e.g. oversampling under-represented groups, discarding stale data). [Challenges continue next page]
• confounding variables – hidden factors influencing variables may result in misleading conclusions, such as home environment stability confounding the relationship between school attendance and educational achievement.
• fairness and equity – predictions often reflect the biases within the data and the wider system where the data originates from; as it is difficult to satisfy all the different definitions of fairness, a normative decision is required for what kind of equity is desired and optimised for.
• accountability – as no human was directly responsible in generating the decision-making process making, responsibility and liability is less clear.

ML models that infer how to generate outputs based on the relationship of previously observed inputs with an associated output.

Subsets: self-supervised learning (associated output comes from the input data, typically used in generating large complex sequences like languages, images)

Examples: classifiers (decision trees, gradient boosting: e.g. StatsNZ International Migration predictions, logistic regression: e.g. ACC Probability of Accept / RoC*RoI), regressions (linear regression)

Evaluation: measured based on how well the model performs on past but previously unseen examples of data.

Challenges:

• generalisation – careful testing methodology required to ensure patterns are learnt (i.e. not just examples memorised) that can be accurately applied outside of training and for extreme edge cases.
• validity of targets – assumes target outputs are correct and consistent, such as adjusting for environmental or legislative changes since the data was captured
• suitability of targets – critically assessing whether the chosen predicted output (e.g. risk of reimprisonment and reconviction) aligns with broader desired outcomes (e.g. releasing an offender will not result in future societal adversities), where the output variable may introduce confounding (e.g. reimprisonment may be less likely in less policed areas).

ML models that infer how to generate outputs from the input it receives without explicitly knowing what outputs to generate (from past examples), based only on underlying patterns in the input data. ⁶

Subsets: semi-supervised learning (a small, labelled dataset provides the starting point for unsupervised learning on a larger unlabelled dataset)

Examples: anomaly detection (e.g. fraud and abuse detection), clustering (e.g. cohort detection for targeted policy interventions)

Evaluation: measured by how well the model captures structure and patterns in the data (e.g. separation of clusters, reconstruction quality). Developing robust objective metrics is more difficult during model development because, unlike supervised learning, there is no known ground truth. A semi-supervised approach (labelling a small dataset) may be used for known concepts like fraud and abuse.

Challenges:

• fairness and bias – despite seeming inherently unbiased by learning from underlying patterns, it is still susceptible to learning systemic biases embedded in the input data, or statistical biases from a lack of sufficient representation.
• imbalanced data – patterns of interest (e.g. fraud and abuse) may be overshadowed by patterns of normal behaviour, making detection or pre-definition difficult.

ML models that infer how to generate outputs through reproducing input data rather than relying on an independently provided target. As the target output comes from the input data itself, this ML paradigm combines the verifiability of supervised learning (withheld data can be used as ground truth) and the scalability of unsupervised learning (no manual labelling and eliminates bias from proxy labels).

Examples: LLM pre-training, large population models (e.g. PHF Science’s ALMA)

Evaluation: measured by how well the model recreates previously unseen input data.

Challenges: fairness and bias as with unsupervised learning

DL models that create new data (text, images, video, music, speech) by learning and mimicking the underlying patterns within existing data.

Examples: large language models (e.g. GPT-5, Claude Sonnet), orchestrating LLMs (e.g. Microsoft Copilot Analyst, Github Copilot), diffusion models (e.g. Midjourney, OpenAI Sora)

Evaluation: as with all ML techniques, but output evaluation is much more difficult due to the near-infinite and unpredictable nature of its output. The tasks these models have been trained for (e.g. generally predicting the next likely word in a sequence) may be different from the tasks they are used for in practice (e.g. analysing policy intervention options, performing calculations on tables pasted into chat). Outcome evaluation is therefore more important and is typically done manually (and is always manual when using off-the-shelf models that the deployer does not train further).

Challenges:

• hallucination – production of plausible but false content, exacerbated by the potential for deceptive confidence: delivering false information as authoritatively and fluently as faithful information; and the subtlety of potential errors given its optimisation for linguistic likelihood rather than factuality.
• homogenisation – generating overly uniform content leading to the marginalisation of underrepresented ideas and identities by regressing to the mean.
• user hijacking – manipulating the system with inputs that result in undesirable behaviour (e.g. prompt injection, data poisoning)
• supply chain provenance – lack of transparency around the reliability of constituent components of a GenAI system (e.g. are datasets and pre-trained models accurate, representative, and obtained legally and ethically).
  • Provenance is especially important if outputs may derive from:
    • protected mātauranga Māori (e.g. language, art, knowledge) as permission must be sought from the authors or kaitiaki of such materials
    • tapu materials which should never be used to create new outputs
• generation of dangerous content – including dangerous weapons; dangerous, violent or hateful content; misinformation and disinformation; offensive cyber-attacks; obscene harmful imagery
• unauthorised data integration or deanonymisation – leakage of sensitive information from training data or inputs, both explicitly (e.g. publicly available social media profiles) and implicitly (e.g. semantic cues that imply personal attributes)
• content attribution – as part of the general challenge of auditability and recordkeeping.
• environmental impact – training models require vast computational resource (and thus energy or water for cooling); impact is less so for end use of pre-trained models but cumulative resource use can be substantial at scale.

AI models that learn how actors in an environment should effectively act (known as an actor’s policy, e.g. writing the next best word, perform the next best economic transaction, wait or administer a medical treatment) rather than training on past examples.

Examples: large language models (RL from human feedback), macroeconomic models (e.g. Salesforce AI Economist), dynamic healthcare treatment personalised to patient characteristics and needs and real-time diagnostics

Challenges: generalisability – genuinely robust solutions are hard to distinguish over fortuitous solutions performing well under the conditions it encountered in a particular training run. This instability is worsened as each actor’s actions directly shape the data it learns from, creating correlations that amplify noise.

AI models that mimic the trial-and-error, survival-of-the-fittest nature of biological evolution. Unlike traditional machine learning, which primarily optimises solutions mathematically, evolutionary computation randomly adjusts candidate solutions and retains a subset of the best ones. These methods can be especially useful when conventional ML reaches a dead end and cannot find a mathematical way to improve a solution, or when there is no clear mathematical formulation for how to improve it. Evolutionary computation maintains a pool of potentially effective candidates and evaluates them using externally defined measures of “fitness” that are not constrained by the internal structure of the solution.

Examples: actor behaviour in simulations (e.g. MATSim in Ministry of Transport’s Monty traffic model: simulated transport system users determine their own daily travel plans based on historical data, ‘compete’ with other transport system users, everyone reaches an optimised equilibrium that balances everyone’s travel objectives without pre-defining behaviour – beneficial for modelling infrastructure change), resource allocation and scheduling

Challenges: genetic instability – careful model design is often required to ensure beneficial traits are not lost to excessive variation, reducing the likelihood of settling on a truly effective solution.