Tigera Logo

Guides: Generative AI Cyber Security

Generative AI Security: 7 Critical Risks and Defending Your Organization

What Is Generative AI?

Generative AI security protects AI systems and data from threats like prompt injection, data poisoning, and unauthorized access. It involves securing LLMs throughout their lifecycle to prevent data leakage and ensure output integrity. Organizations must implement a robust security policy that covers AI usage, access controls, and data handling.

Generative AI security risks include:

  • Prompt injection: Attackers feed malicious prompts to bypass security filters.
  • Data poisoning: Manipulating training data to corrupt the model’s output.
  • Sensitive data disclosure: GenAI inadvertently leaking sensitive training data in its responses.
  • Shadow ai: Unapproved, unmonitored use of AI tools by employees within an organization.
  • Insecure output: AI-generated code or content that contains vulnerabilities.

Key strategies for securing generative AI:

  • Implementing guardrails: Using tools that monitor inputs and outputs to block malicious or inappropriate content.
  • Adopting security frameworks: Following guidelines like the NIST Artificial Intelligence Risk Management Framework to manage risks.
  • Red teaming: Testing models against adversarial attacks to find vulnerabilities before they are exploited.
  • Data protection and privacy: Ensuring training data is sanitized and that interaction with models doesn’t violate data privacy regulations.
  • Monitoring and evaluation: Continuous auditing of model performance to detect drift and potential security breaches.

This is part of a series of articles about LLM security.

In this article:

How do Generative AI Models Work?

Generative AI models utilize several neural network architectures to achieve their capabilities. State-of-the-art image generation models use the diffusion technique, which involves training a neural network to denoise images blurred with Gaussian noise. Models are trained to reverse the process of adding noise to an image, thus generating new, photorealistic images based on natural language prompts.

Text generation models are typically based on the Transformer architecture, first introduced in a 2017 paper by Google, which underpins today’s advanced large language models (LLMs). Transformers utilize an attention mechanism to manage and predict sequences in data, enhancing their ability to generate coherent and contextually relevant outputs based on the input data they process.

A newer innovation in generative AI is the introduction of multi-modal models, which are trained simultaneously on text, images, and videos, and are able to receive inputs and generate outputs combined of any type of information. The first fully multi-model generative AI model was Google Gemini, which is able to understand and reason about textual, image, or video content.

The Impact of Generative AI Security Risks

The expansion of generative AI introduces several security risks, primarily because it can be used to create deceitful and manipulative content. The potential for generating deepfakes, synthetic identities, and counterfeit documents can lead to fraud, misinformation, and other malicious activities. These capabilities pose a significant threat to personal, corporate, and national security, making the potential abuse of generative AI technologies a critical issue.

Furthermore, as generative AI systems become more integrated into business and governmental operations, the risks extend to the manipulation of these systems themselves. Bad actors could influence AI behavior to cause favorable outcomes, create biased results, or disrupt services. The imperative to secure generative AI is not just about protecting the systems but safeguarding the outputs they generate, which increasingly influence real-world perceptions and decisions.

Related content: Read our guide to AI safety.

What are the Different Types of GenAI Security?

Generative AI security is a broad discipline that covers the protection of AI models, the applications built around them, the data they process, and the content they generate. Because generative AI systems interact with users in open-ended ways, they require security controls that go beyond traditional application security.

The main types of GenAI security include:

1. Model Security

Model security focuses on protecting the underlying AI model from manipulation, theft, or misuse. Foundation models can be valuable intellectual property, and attackers may attempt to extract model behavior, reverse-engineer outputs, or abuse model access through automated queries.

Model security also involves defending against attacks that alter the way the model behaves. For example, an attacker might try to poison training data, fine-tuning data, or retrieval sources so the model produces biased, harmful, or incorrect responses.

2. Prompt and Input Security

Prompt and input security addresses threats that occur when users interact directly with generative AI systems. One of the most common risks is prompt injection, in which a malicious user crafts an input designed to override system instructions, reveal sensitive data, or force the model to take unintended actions.

This category also includes jailbreak attempts, malicious file uploads, hidden instructions in documents or webpages, and adversarial inputs that attempt to bypass safety controls. Organizations using GenAI must validate, filter, and monitor inputs before they are sent to the model.

3. Data Security and Privacy

Generative AI systems often process sensitive business data, personal information, source code, financial records, or customer communications. Data security ensures that this information is not exposed to unauthorized users, stored insecurely, or used in ways that violate privacy requirements.

Important controls include data loss prevention, access control, encryption, retention policies, and careful management of whether user inputs are used for training or model improvement. Enterprises must also ensure that AI tools do not accidentally reveal confidential information in generated responses.

4. Output Security

Output security focuses on the content produced by generative AI systems. Even when a model is not compromised, it may generate inaccurate, harmful, biased, copyrighted, or unsafe content. In enterprise settings, insecure outputs can lead to legal risk, reputational damage, or operational errors.

Security teams can reduce these risks by applying response filtering, grounding outputs in trusted sources, requiring human review for high-risk use cases, and monitoring for hallucinations or policy violations.

5. Application and API Security

Many GenAI systems are embedded into applications, chatbots, copilots, developer tools, and automated workflows. These applications commonly connect AI models to databases, APIs, plugins, file systems, and business systems. This creates a new attack surface.

Application and API security ensures that AI-powered applications follow secure design principles. This includes authentication, authorization, rate limiting, logging, secure plugin design, and strict control over what actions an AI agent is allowed to perform.

6. Infrastructure and Runtime Security

Generative AI workloads often run on complex infrastructure, including cloud environments, Kubernetes clusters, GPUs, model-serving platforms, vector databases, and data pipelines. Infrastructure security protects these environments from compromise.

This includes securing containers, networks, workloads, secrets, APIs, and model endpoints. Runtime monitoring is especially important because AI applications may behave dynamically and interact with external systems in ways that are difficult to predict.

7. AI Supply Chain Security

Generative AI systems depend on a broad supply chain of models, datasets, libraries, APIs, plugins, and open source components. A vulnerability or malicious component anywhere in this chain can affect the security of the entire system.

AI supply chain security includes verifying model provenance, scanning dependencies, securing training and fine-tuning pipelines, validating datasets, and monitoring third-party AI services for risk.

8. Governance, Compliance, and Responsible AI

Governance ensures that generative AI is used safely, legally, and ethically across the organization. This includes defining policies for acceptable use, human oversight, auditability, risk assessment, and regulatory compliance.

Responsible AI practices also address fairness, transparency, explainability, and accountability. While these areas are not purely technical, they are essential to reducing the broader security and business risks introduced by generative AI.

Generative AI Security Risks

While generative AI has tremendous potential in improving human productivity, it also raises new and severe security risks. Here are some of the important risks that must be considered by individuals or organizations developing, deploying, or using generative AI technology.

1. Deepfakes

Deepfakes represent a significant challenge in cybersecurity, stemming from generative AI’s ability to create convincing fake images and videos. These can be used to impersonate individuals in phishing attacks or to spread misinformation. The realism of deepfakes makes it difficult for both individuals and traditional security systems to detect falsifications, increasing the risk of security breaches.

Moreover, the propagation of deepfakes can undermine trust and credibility in digital communications. As generative AI continues to improve, the sophistication and indistinguishability of deepfakes are likely to increase, posing elevated threats to personal and organizational security.

2. Prompt Injection

Prompt injection occurs when malicious inputs are provided to an AI system, leading it to generate outputs that facilitate a security breach. This form of attack can alter the behavior of AI-powered systems, causing them to malfunction or produce harmful outputs. Detecting such manipulation can be challenging, as it requires constant vigilance over the inputs and outputs of AI systems.

For organizations using generative AI, safeguarding against prompt injection involves rigorous input validation and continuous monitoring of AI activities. Early detection and response are crucial to mitigate potential damages and maintain system integrity.

3. Data Poisoning

Data poisoning is a tactic where attackers corrupt the training data of AI systems, leading to compromised model integrity. By inserting false or misleading data into the training set, attackers can skew the AI’s output or cause it to fail under specific conditions. This manipulation can seriously affect decision-making processes and operational effectiveness.

To combat data poisoning, cybersecurity professionals must implement stringent data validation and cleansing processes. Ensuring the accuracy and reliability of training data is vital to maintaining the effectiveness and trustworthiness of generative AI applications.

4. Exploitation of AI Biases

Generative AI systems can inherit or develop biases based on the data they are trained on, which can be exploited by attackers. For example, if an AI system disproportionately associates certain behaviors or traits with specific demographics, it could lead to skewed security measures against those groups. Exploiting these biases can enable attackers to engineer specific scenarios that the AI fails to recognize or mishandles.

Addressing AI biases involves regular audits, diverse data sets for training, and the implementation of fairness algorithms to detect and mitigate biased outcomes. Maintaining ethical standards and fairness in AI operations is critical to its success and acceptance in cybersecurity roles.

5. Model Theft

Model theft is a growing concern in the context of generative AI in cybersecurity. Attackers may steal AI models to understand their structure and functioning, which can be exploited to bypass security mechanisms. Stolen models can also be replicated or sold illegally, undermining the competitive advantage of original developers.

Protecting AI models involves securing data pipelines and storage, implementing robust access controls, and continuously monitoring for any unauthorized access. As AI models become more integral to business operations, ensuring their security against theft is crucial. This includes implementing best practices for container security to protect the environments where proprietary models are trained and deployed.

Model theft is not limited to developers of foundation models. Many organizations are fine-tuning and training generative AI models on their own data. These custom models are a new type of intellectual property, which represents a valuable target for attackers.

6. Privacy Risks

Privacy risks arise when generative AI handles sensitive personal or organizational data. There are concerns about how data is used, stored, and accessed within AI systems, especially considering that breaches could expose vast amounts of confidential information. Ensuring data privacy is paramount to maintaining trust and compliance with regulations like GDPR.

To secure privacy, organizations providing AI services or deploying them for their customers and employees must enforce strict access controls, data encryption, and anonymization techniques where possible. Comprehensive privacy policies that address the specifics of generative AI are essential to safeguard sensitive information against unauthorized access or leaks.

7. Shadow AI

Shadow AI refers to the unauthorized or unmanaged use of generative AI tools within an organization. Employees may use public AI platforms to improve productivity, analyze data, summarize documents, or generate content without formal approval from IT, security, or compliance teams. While often well-intentioned, this creates serious security risks because sensitive data, intellectual property, customer information, or internal business documents may be entered into systems that are not properly governed.

The risks of Shadow AI include data leakage, regulatory non-compliance, lack of auditability, inconsistent security controls, and exposure of confidential information to third-party AI providers. Organizations may also lose visibility into how AI-generated outputs are being used in business processes, which can introduce errors, bias, or unverified information into critical workflows.

To address Shadow AI, organizations should establish clear AI usage policies, provide approved and secure AI tools, educate employees on safe AI practices, and monitor for unauthorized AI usage where appropriate. Rather than simply blocking AI tools, organizations should create governed alternatives that allow employees to benefit from generative AI while maintaining security, privacy, and compliance.

Notable Generative AI Security Frameworks

OWASP Top 10 for LLM Applications

The OWASP Top 10 for LLM Applications is one of the most widely used security frameworks for identifying and mitigating risks in large language model applications. It focuses specifically on vulnerabilities that arise when LLMs are integrated into software systems, APIs, enterprise workflows, and autonomous agents.

The framework identifies the most critical categories of LLM-related risk, including prompt injection, sensitive information disclosure, supply chain vulnerabilities, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation, and unbounded consumption. These risks are especially relevant for organizations building AI-powered chatbots, copilots, retrieval-augmented generation (RAG) systems, and AI agents that can access tools or sensitive enterprise data.

For security teams, the OWASP Top 10 provides a practical starting point for threat modeling generative AI applications. It helps teams evaluate how user inputs are handled, how model outputs are validated, what permissions AI systems have, and whether sensitive data can be exposed through prompts, responses, logs, or connected tools. The framework is particularly valuable because it translates AI security concerns into familiar application security practices, making it easier to integrate LLM security into secure software development lifecycles.

NIST AI RMF and Generative AI Profile (NIST AI 600-1)

The NIST AI Risk Management Framework (AI RMF) provides a structured approach for identifying, assessing, and managing risks across the AI lifecycle. Its core functions—govern, map, measure, and manage—help organizations create repeatable processes for evaluating AI systems, assigning accountability, and reducing risks before and after deployment.

NIST’s Generative AI Profile, NIST AI 600-1, adapts the broader AI RMF to the specific risks introduced or amplified by generative AI. These include risks related to synthetic content, hallucinations, cybersecurity misuse, privacy leakage, intellectual property exposure, bias, harmful content generation, and overreliance on AI-generated outputs. The profile was developed as a companion resource to the AI RMF and is intended to help organizations apply risk management practices to generative AI systems in a sector-agnostic way.

For enterprises, the NIST framework is especially useful for governance and compliance. It encourages organizations to document AI use cases, define risk tolerances, evaluate impacts on users and stakeholders, monitor systems continuously, and implement controls appropriate to the level of risk. In generative AI deployments, this can include red teaming, model evaluation, content provenance, human oversight, access control, incident response processes, and policies for acceptable AI use.

Google Secure AI Framework (SAIF)

Google’s Secure AI Framework (SAIF) is a security framework designed to help organizations build, deploy, and operate AI systems securely. It extends traditional security principles to the AI ecosystem, recognizing that AI systems introduce risks across data, infrastructure, models, applications, and user interactions.

SAIF is organized around six core elements: expanding strong security foundations to the AI ecosystem, extending detection and response to AI-related threats, automating defenses, harmonizing platform-level controls, adapting controls to improve feedback loops, and contextualizing AI system risks within surrounding business processes.

In practice, SAIF encourages organizations to secure the entire AI development and deployment pipeline. This includes protecting training data, securing model artifacts, hardening infrastructure, monitoring AI behavior, limiting access to sensitive systems, and applying security controls consistently across AI-enabled applications. SAIF is particularly relevant for organizations adopting AI at scale, because it emphasizes operational security, continuous monitoring, and shared responsibility across security, engineering, data science, and governance teams.

MITRE ATLAS

MITRE ATLAS, short for Adversarial Threat Landscape for Artificial-Intelligence Systems, is a knowledge base of tactics, techniques, and procedures used by adversaries against AI-enabled systems. It is modeled in part after MITRE ATT&CK, but focuses specifically on threats to machine learning and AI systems.

ATLAS documents adversarial behaviors across the AI lifecycle, including reconnaissance, resource development, initial access, model access, data poisoning, evasion, model manipulation, exfiltration, and impact. It also includes real-world case studies, mitigations, and mappings that help defenders understand how attacks against AI systems may unfold in practice. MITRE describes ATLAS as a living knowledge base built from real-world observations and demonstrations by AI red teams and security researchers.

For cybersecurity teams, MITRE ATLAS is useful for AI-specific threat modeling, red teaming, detection engineering, and incident response. It helps organizations move beyond abstract AI risks and analyze concrete attacker behaviors, such as stealing model artifacts, probing model behavior, poisoning training data, extracting sensitive information, or manipulating AI outputs. When combined with frameworks like OWASP, NIST AI RMF, and SAIF, MITRE ATLAS provides a practical adversarial perspective on how generative AI systems can be attacked and defended.

Enterprise Best Practices for Generative AI Cybersecurity

Here are a few steps organizations can take to prevent and mitigate threats when using generative AI technology.

Input Validation

Input validation is essential to safeguard generative AI systems from manipulative or harmful inputs that could compromise system integrity. By rigorously checking data before it is processed, organizations can prevent unwanted or malicious content from influencing AI behavior. This practice is particularly important in environments where AI interacts directly with users or other systems.

Implementing comprehensive input validation protocols enhances the resilience of AI systems against attacks such as data poisoning and prompt injections. It contributes to the overall security architecture, ensuring that only clean, verified data influences AI decisions and functions.

Maintain Clear Documentation of AI Model Development and Deployment

Maintaining clear documentation of AI model development and usage is fundamental for auditability and continual improvement. This documentation should include details of the data used, model training processes, versioning information, performance metrics, use cases, and information about model fine-tuning. Accurate records ensure that AI systems are transparent and their functions fully understood by developers, operators, and auditors alike.

Such rigorous documentation practices not only help in maintaining operational consistency but also assist in compliance with legal and regulatory requirements. It serves as a basis for refining AI models, troubleshooting issues, and enhancing system reliability.

AI-Specific Incident Response Plans

AI-specific incident response plans are tailored to address potential issues arising from the deployment of AI technologies. These plans include scenarios like data breaches, system misconfigurations, and AI model failures. Having a specialized response mechanism ensures that teams can quickly and efficiently handle AI-related incidents, minimizing negative impacts.

These plans should integrate with broader cybersecurity strategies, providing a cohesive response across all technological fronts. They play a crucial role in maintaining service continuity, safeguarding data, and protecting the integrity of AI systems.

Protect Sensitive Data and Prevent Data Leakage

Organizations should implement strict controls to prevent sensitive, confidential, regulated, or proprietary data from being exposed through generative AI systems. This includes applying data loss prevention controls, masking or redacting sensitive inputs, limiting what data can be sent to external AI providers, and preventing models from revealing secrets, credentials, customer data, or internal system information.

OWASP highlights sensitive information disclosure as a core LLM application risk, especially when models are connected to internal data sources or enterprise tools. Organizations should define clear rules for what data can be used in prompts, training, fine-tuning, retrieval-augmented generation, and AI-generated outputs.

Implement Strong Access Controls and Least Privilege

Generative AI systems should operate under strict access controls. Users, applications, agents, and AI tools should only have the permissions necessary to perform their intended tasks. This is especially important for AI agents that can access files, databases, email, development environments, ticketing systems, or external APIs.

Organizations should enforce identity and access management, role-based access control, multi-factor authentication, and privilege separation for AI-connected systems. Least privilege reduces the impact of prompt injection, account compromise, malicious automation, and unauthorized data retrieval.

Secure the AI Supply Chain

Enterprises should assess and monitor the full AI supply chain, including model providers, open-source models, training data, fine-tuning datasets, APIs, plug-ins, software libraries, cloud infrastructure, and third-party integrations. AI supply chain risk can arise from compromised dependencies, poisoned datasets, insecure model repositories, vulnerable plug-ins, or unverified vendor claims.

Organizations should use vendor risk assessments, software bills of materials, model provenance tracking, dependency scanning, contractual security requirements, and continuous monitoring to reduce supply chain exposure.

Continuous Monitoring, Red Teaming, and Security Testing

Generative AI systems should be continuously monitored and tested before and after deployment. Security teams should conduct AI-focused red teaming, adversarial testing, prompt injection testing, abuse-case testing, jailbreak testing, and monitoring of model outputs for unsafe, inaccurate, or policy-violating behavior.

NIST’s Generative AI Profile recommends managing generative AI risks through structured risk identification, measurement, management, and governance. Continuous testing helps organizations detect weaknesses before attackers exploit them and ensures that security controls remain effective as models, prompts, data sources, and integrations change over time.

AI Security with Calico

Calico offers numerous features to address the many network and security challenges faced by cloud platform architects and engineers when deploying genAI workloads in Kubernetes. Here are five Calico features for container networking and security for GenAI workloads:

  1. Egress access controls – Calico provides granular, zero-trust workload access controls between individual pods in Kubernetes clusters to external resources such as LLMs. It provides fine-grained workload access controls using DNS egress policies and NetworkSets (using IPs/CIDRs in network policy).
  2. Egress gateway – The Calico Egress Access Gateway assigns a fixed, routable IP to a Kubernetes namespace. All egress pod traffic from that namespace with an assigned routable IP address identifies the workload running within that namespace. This enables the cluster to securely scale while preserving the limited number of routable IPs and leveraging non-routable IPs for all other pod traffic within the cluster. The Calico Egress Gateway works with any firewall, enabling Kubernetes resources to access endpoints behind a firewall securely.
  3. Identity-aware microsegmentation – Calico enforces microsegmentation to achieve workload isolation and secure lateral communication between pods, namespaces, and services. It enables teams to logically divide workloads into distinct security segments and then define granular security controls for each unique segment. Teams can isolate workloads based on environments, application tiers, compliance needs, user access, and individual workload requirements.
  4. Observability and troubleshooting – Calico’s Dynamic Service and Threat Graph provides a graph-based visualization of your Kubernetes deployments, including images, pods, namespaces, and services. It has built-in troubleshooting capabilities to identify and resolve security and compliance gaps, performance issues, connectivity breakdowns, anomalous behavior, and security policy violations.
  5. Cluster mesh – Calico provides a centralized, multi-cluster management plane to enable security, observability, and advanced networking for workloads and services across multiple clusters in hybrid and multi-cloud environments. Calico provides unified security policy controls and federated endpoints and services.

Next steps:

X