Agentic AI has the potential to change how we as professionals carry out our jobs and how businesses organise and deploy their workforces. Imagine a world where people work alongside autonomous AI agents every day. They may work either one-to-one, in close collaboration, or at scale, with agents handling complex business processes and policies.
This world isn’t very far away. In fact, businesses are already preparing for this eventuality with in-house developers building new AI systems to support the rollout of AI-powered agents.
However, this does carry an element of risk. Businesses can’t afford situations where agents inadvertently misinterpret instructions or goals, perform tasks incorrectly or cause offence, especially when they are customer-facing. These issues would impact operations and potentially even lead to reputational damage. As a result, software development and quality assurance (QA) teams are building new fail-safes into their testing and QA processes to prevent failures from occurring.
One of the techniques they have started to employ is a process known as red team testing or ‘red teaming.’ Originally developed for cybersecurity, traditional red teaming involved a team of experts simulating attacks on an organisation’s cyber defences to identify vulnerabilities.
Also known as adversarial testing, this method has since been adopted to test intelligent systems like Gen AI and now agentic AI. It’s used to identify problems related to security, safety, accuracy, quality, functionality or performance. Red teaming can unearth points of failure that would be difficult to identify through regular testing methods.
Agentic AI rips up the rule book
Agentic AI represents a major shift in how software systems operate. Unlike traditional AI models or rule-based automation, agentic AI systems are designed to pursue goals autonomously, making real-time decisions using planning, memory, tool use, and feedback loops. This distinction means that developers and QA professionals must alter their approach to software testing.
Traditional software follows a deterministic path; it produces output B when given input A. Instead of following hard-coded rules, agentic AI systems interpret high-level goals (e.g., “update my contact information” or “book the cheapest flight to London”) and decide how to achieve them.
This autonomy introduces a great amount of flexibility where agents can solve problems in novel ways, but it also introduces non-determinism. The actions agents take depend on factors such a current context, prior memory, available tools and how well they understand a specific task.
As a result, the same input may lead to different behaviour depending on subtle environmental changes. This unpredictability has meant that developers and QA professionals now have to test and debug intent, strategy and behaviour, not just code correctness. As a result, new evaluation methods and specifically red teaming come into play.
AI red teaming in action
Red teams have been a part of Gen AI for years now. For example, Microsoft has its own AI red team that has reportedly tested over 150 Gen AI systems across Microsoft since it was formed in 2018.
Agentic AI red teaming involves a systematic adversarial approach employed by human testers to identify issues in AI models and solutions. Once those issues have been identified, the information is passed on to developers. Developers will then have the scope to retrain the underlying AI models or develop specific guardrail rules to mitigate any risk.
The red team’s role is typically to identify systemic issues or to focus on goals such as security and safety, or on domain-specific topics. While some testing can be done regardless of a human tester’s background, the best practice is to recruit based on domain specialist knowledge or demographic characteristics.
Domain specialists can be brought on for their deeper knowledge of specific subjects, such as financial services, to test agents designed to support digital banking. Their expertise complements testers on the team who have more generalist characteristics. Their input would benefit mainstream consumer brands like banks or retailers, helping them to better understand how an agent might interact with a broad user base.
Businesses would prefer to surface potential issues during testing before they arise as customer complaints in production.
The original cybersecurity red team also has a crucial place in Agentic AI testing.
Agents provide a new attack vector for malicious actors, which means security testing is paramount. Red teaming can help organisations keep pace with the rapidly evolving threat landscape of AI systems. Cybersecurity Red teams teams can identify vulnerabilities but also coordinate with other stakeholders to propose viable security solutions that will add resilience to the software development lifecycle.
The role of the red team
AI systems continue to demonstrate ever more complex behaviours and autonomous capabilities. Testing and validation methodologies must be able to keep pace with these changes, without hindering the agents’ potential. AI red teaming has become an essential component of software testing and QA processes.
Especially now with businesses under so much pressure to quickly iterate and deploy AI systems for agentic AI and other use cases. Therefore, it is essential that humans remain in the loop to ensure that agentic AI systems evolve safely and securely.
Red teaming methodologies and other evaluation frameworks should be embedded into the development lifecycles of organisations that are building and deploying AI systems. Organisations can implement effective red teaming by selecting from a diverse pool of individuals and groups.
These diverse teams should provide a blend of domain expertise and demographic characteristics. This mix is crucial because the quality of a red team’s output relies heavily on the quality of the testing team.
Whether your team is managing red teaming internally or outsourcing to a testing partner, having AI red teaming experts on hand, plus a large and diverse community of testers, helps ensure your approach is comprehensive, mitigating risks and preserving customer loyalty and brand image.
https://www.enterprisetimes.co.uk/2025/11/17/agentic-ai-testing-gets-adversarial-with-ai-red-teaming/a>