Forem: Gelu Vac

Security by Design in the Age of Artificial Intelligence: Fundamentals, Risks and Resilience Strategies

Gelu Vac — Mon, 13 Apr 2026 14:49:33 +0000

We live in a time when artificial intelligence has moved from being an emerging technology to a critical infrastructure. Algorithms decide what information we see, what transactions are suspicious, what diagnoses are likely, and increasingly, what decisions are optimal for organizations and governments. Companies like OpenAI, Google DeepMind, and Anthropic have accelerated the development of advanced models, making artificial intelligence a central element of the digital economy. In this context, security can no longer be an afterthought or a reactive mechanism. It must be an integral part of the system architecture from the moment of its conception.

Over the past two decades, digital transformation has been accelerated by cloud, mobility, and big data. In recent years, however, artificial intelligence (AI) has become the main driver of innovation.
But this revolution comes at a cost: the attack surface is growing exponentially.

Security by Design is not just about installing additional controls or enforcing stricter access policies. It means deeply integrating security principles into the technological DNA of a system. In the AI era, this approach becomes vital, because the complexity of models, their dependence on massive data, and the often opaque nature of algorithms create an environment in which vulnerabilities can be subtle but devastating.

The concept of Security by Design (SbD) involves integrating security from the design phase of a system, not as a later stage or a “patch” after vulnerabilities appear. In the AI era, this approach becomes critical because:

AI models are dependent on massive data.
Algorithms can be manipulated.
Automated decisions can have a major impact on users.
AI systems can be exploited in new ways (prompt injection, model inversion, data poisoning).

Security by Design in the AI era means responsible design, technical resilience, and solid governance.

From traditional security to algorithmic security

The concept of Security by Design is not new. It has been promoted since the 1990s in the field of application security and IT infrastructure. It initially appeared in the field of software security and IT infrastructures, being supported and formalized by organizations such as the National Institute of Standards and Technology (NIST) that developed frameworks such as the NIST Cybersecurity Framework, which promotes the integration of security into all stages of the life cycle. The idea was simple: it is more efficient and safer to prevent vulnerabilities by design than to correct them after the system has been compromised.

Classic principles include:

Principle of least privilege
Defense in depth
Fail secure
Zero Trust

In traditional systems, security focused on perimeter protection, authentication, encryption and vulnerability management. In the case of artificial intelligence, however, the object of protection is no longer only the infrastructure, but also the model itself. The algorithm becomes a critical asset. Training data becomes an attack surface. Automated decisions can have major legal and ethical implications.

Thus, security must be extended beyond servers and networks, to the algorithmic and epistemic level of the system, respectively, in the AI context, traditional principles must be extended to cover:

Training data security
Model integrity
Robustness against adversarial attacks
Explicability and auditability

Vulnerabilities specific to the AI era

Data poisoning

An AI system can be compromised not only through unauthorized access, but also through manipulation of the data on which it relies. Data poisoning attacks, in which malicious data is introduced into the training set, can subtly alter the behavior of the model. The result is not an obvious error, but a strategic degradation of performance or an intentional deviation of decisions.

Data poisoning is a type of attack in which the adversary compromises the integrity of an artificial intelligence model by deliberately introducing malicious data into the training set. Unlike attacks that target the system after implementation, data poisoning acts “at the source”, affecting the learning process of the model and influencing its long-term behavior. The injected data can be designed either to degrade the overall performance (unavailability or decreased accuracy) or to create a “backdoor” so that the model reacts erroneously only in the presence of a specific pattern or trigger. For example, a spam detection system could be trained with manipulated messages so that certain malicious expressions are later considered legitimate. The vulnerability is amplified in scenarios where the data comes from open, collaborative or automatically collected sources, without rigorous validation. From a Security by Design perspective, preventing data poisoning involves strict verification of the provenance of the data, auditing of datasets, anomaly detection mechanisms and controlled separation of data streams used for training.

Adversarial Attacks

Adversarial attacks are another sophisticated category of threats in which an adversary subtly manipulates the input data of an artificial intelligence model to cause systematic classification or decision errors, without the changes being obvious to human users. In the case of computer vision models, for example, adding minimal perturbations - invisible to the naked eye - to an image can cause the system to misidentify an object (a traffic sign can be misclassified, with serious implications for autonomous vehicles). Similarly, in natural language processing, the insertion of specially constructed linguistic structures or tokens can skew the interpretation of the model. These attacks exploit the mathematical sensitivity of neural networks to small variations in the multidimensional data space, demonstrating that high performance on standard data does not guarantee robustness under adversarial conditions. From a Security by Design perspective, countermeasures include adversarial training, robust input validation, and monitoring for abnormal model behavior in production.

Model Inversion & Data Extraction

Model Inversion is an attack technique in which an adversary attempts to reconstruct sensitive information about the data used to train a model, using only access to the final (black-box or sometimes white-box) model. The central idea is that machine learning models “retain” to some extent statistical characteristics of the training data. If the model is strategically interrogated, the attacker can approximate individual data or sensitive features associated with a particular user. For example, in a facial recognition system or a medical model trained on clinical data, an attacker could reconstruct facial features or information about a specific patient. The vulnerability arises especially when the model is overfitted or when its responses provide detailed probability scores, which can be exploited for reverse inference.

Data Extraction (or model extraction / training data extraction) goes even further, aiming at directly extracting fragments of data stored by the model during training. In the case of large language models, this can mean regenerating portions of sensitive texts accidentally included in datasets (personal data, API keys, confidential information). Attackers use iterative queries, strategic formulations or optimization techniques to “squeeze” the model of stored information. The risk is amplified when the models are integrated into public applications and provide very detailed answers. From a Security by Design perspective, protection against these attacks involves limiting the granularity of the outputs, using regularization and differential privacy techniques in the training phase, as well as implementing robust mechanisms for monitoring and filtering the generated answers.

Prompt Injection (in LLMs)

Unlike traditional attacks on web applications or infrastructure, prompt injection does not exploit a classic programming bug, but the very probabilistic and contextual nature of the model.

Prompt Injection is a technique by which a user inserts malicious instructions into an apparently legitimate input, with the aim of modifying the behavior of the model and causing it to ignore the rules or restrictions established by the system.

Typically, a system based on LLM works as follows:

There is a system prompt (internal instructions, invisible to the user).
There is a user prompt (user input).
The model generates a response based on the entire context.

The vulnerability arises because the model treats all text as a sequence of tokens, without making a rigid structural distinction between system and user instructions. Thus, a user can try to "overwrite" the initial rules with a specially constructed input.

Security by Design in the AI era involves anticipating these scenarios and designing the system so that it is robust, resilient and able to detect behavioral deviations.

Integrating Security into the AI Lifecycle

An AI system goes through several stages: data collection, model training, validation, deployment, and ongoing operation. Each of these phases should be treated as a critical control point.

In the data collection phase, security means verifying sources, ensuring the integrity of datasets, and protecting sensitive data through anonymization or pseudonymization. Data is the foundation of the model; if this foundation is compromised, the entire system becomes fragile.

In the training phase, the infrastructure must be isolated and monitored. Models must be versioned, and each experiment must be documented to allow for later auditing. The integrity of generated artifacts must be verified through cryptographic mechanisms, and access to computing resources must be limited according to the principle of least privilege.

At the time of deployment, API security becomes essential. Rate limiting, multifactor authentication, and monitoring for abnormal behavior are measures that reduce the risk of system exploitation. In the operational phase, continuous monitoring of model performance and detection of model drift are essential to prevent degradation or manipulation of its behavior.

Security by Design therefore means a continuity of control, not a single event.

Zero Trust and the new paradigm of trust

In the AI era, the concept of trust must be rethought. The Zero Trust model, which assumes that no user or system is implicitly trusted, becomes extremely relevant. Access to AI models and associated data should only be granted based on clear and verifiable policies.

This approach is all the more important as AI systems are integrated into complex ecosystems, distributed in the cloud, and connected to multiple data sources. The lack of segmentation or granular access controls can turn a minor incident into a major breach.

Zero Trust applied to AI is not limited to authentication; it also involves continuous validation of system behavior, verification of model integrity, and permanent analysis of user interactions.

Regulation, governance and accountability

As AI becomes critical infrastructure, regulation becomes inevitable. The European Union has introduced the AI Act, which classifies AI systems according to risk and imposes strict requirements for those considered high-risk. In parallel, the GDPR establishes clear obligations on data protection and the right to explanation.

These legislative frameworks are not obstacles to innovation, but catalysts for the adoption of Security by Design. They force organizations to document processes, implement audit mechanisms and ensure transparency.
AI governance thus becomes a central element of security. It is not enough for a system to perform; it must be accountable, explainable and compliant with legal norms.

The ethical dimension of security

Security in the age of AI is not just a technical issue. It is also an ethical one. A model that discriminates or produces biased results can generate damage as serious as a data breach.

Companies like Microsoft and IBM have developed Responsible AI frameworks that include principles of fairness, transparency, and accountability. These initiatives show that Security by Design must also include protection against social and moral risks.

Ultimately, security is not just about protecting the system, but also protecting the people affected by its decisions.

Red Teaming and Operational Resilience

A secure system is not one that has not been attacked, but one that has been rigorously tested and proven resilient.

Red Teaming in the context of AI systems is a structured process through which specialized teams simulate real-world attacks to identify vulnerabilities before they are exploited in the operational environment, directly contributing to increasing the resilience of the system. Unlike traditional testing, red teaming involves a creative adversarial approach, in which experts try to circumvent model restrictions through prompt injection, adversarial attacks, data exfiltration attempts, or manipulation of algorithmic behavior. The goal is not just to discover specific technical errors, but to assess the ability of the entire architecture - model, infrastructure, access controls, and organizational processes - to withstand real-world pressures. By integrating red teaming into the continuous development and operations (MLOps) cycle, organizations can transform security from a reaction to incidents into a proactive mechanism for strengthening operational resilience, ensuring the safe and stable operation of AI systems in dynamic and potentially hostile conditions.

This practice transforms security from a defensive to a proactive process. Instead of reacting to incidents, organizations anticipate and model risk scenarios.

Future Challenges

As AI evolves towards autonomous systems and agents capable of making complex decisions independently, the attack surface will continue to grow. The integration of AI into critical infrastructure, financial systems, or healthcare systems will amplify the potential impact of vulnerabilities.

Security by Design must evolve with technology. Collaboration between engineers, security experts, lawyers, ethicists, and policymakers will be required. Without this interdisciplinary approach, the complexity of AI systems may outstrip our ability to control them.

Conclusion: Security as the Foundation of Trust

In the age of artificial intelligence, security is no longer a technical detail, but the foundation of digital trust. Without Security by Design, AI systems can become vulnerable, manipulable, and potentially dangerous tools. With Security by Design, they can become catalysts for progress, supporting innovation in a responsible and sustainable way.

Building secure AI is not about slowing down development, but making it sustainable. In a world where algorithms increasingly influence reality, security becomes the invisible architecture that supports the future.

Bibliography

Pearlson, Keri & Novaes Neto, Nelson. „What is Secure-by-Design AI?”
- Definition and framework for integrating security as a basic principle in the design of AI systems.
ETSI EN 304 223 - Securing Artificial Intelligence (SAI)
- Security technical standard for AI systems, which includes "secure design" principles in the different stages of the AI lifecycle.
UK Government - Code of Practice for the Cyber Security of AI
- Best practices guide for designing and operating AI securely, including the principle of secure design.
Prasad, Anand. „A Policy Roadmap for Secure by Design AI: Building Trust Through Security-First Development”
- Article discussing the need to shift the AI security paradigm from reactive to proactive.
Abbas, Rianat et al. „Secure by design - enhancing software products with AI-Driven security measures”
- Study addressing the integration of AI-based security measures within Secure by Design.

Deliberate Hybrid Design: Building Systems That Gracefully Fall Back from AI to Deterministic Logic

Gelu Vac — Mon, 06 Apr 2026 13:52:12 +0000

In the last decade, Artificial Intelligence has moved from experimental labs into the backbone of mainstream products. From recommendation systems and fraud detection to code assistants and autonomous vehicles, machine learning (ML) models now influence critical decisions at scale. Yet as powerful as these systems are, they are also inherently probabilistic and occasionally unreliable. Anyone who has wrestled with a large language model (LLM) that confidently produces incorrect information knows this firsthand.
This unpredictability is not a flaw - it’s a feature of statistical models. But it does create a serious engineering challenge: how do we build dependable products on top of inherently uncertain components?
The answer, increasingly, lies in a design philosophy that is both pragmatic and strategic: deliberate hybrid design. Rather than treating AI as a standalone “brain,” we integrate it with deterministic, rule-based components - and ensure there is a graceful fallback path when AI fails, is uncertain, or is not required.
This article explores how to apply this philosophy in real-world software systems, why it matters, and how to design AI-enabled solutions that are robust, explainable, and trustworthy.

The Myth of AI Supremacy

The rise of AI has created a strong narrative: if Machine Learning can outperform humans on a task, why not let it run everything? But this mindset - “AI-first” or “AI-only” - can be dangerous. It leads to brittle architectures, opaque decision-making, and poor user experiences.
Consider a few examples:

Autonomous driving: State-of-the-art perception systems detect objects and predict trajectories. But when conditions are unclear - fog, sensor failure, or conflicting signals - the system must hand over control or switch to rule-based safety protocols.
Fraud detection: Machine learning models flag suspicious transactions with impressive accuracy. Yet final decisions often depend on deterministic business rules (e.g., legal compliance thresholds) or require human review.
Chatbots and copilots: Generative AI can draft messages or code snippets. But when the confidence score is low or the consequences are high (e.g., financial transactions, medical recommendations), the system should defer to validated templates or manual input.

In each case, the most reliable solutions are hybrid - combining probabilistic inference with deterministic control. This isn’t a fallback of weakness; it’s a design choice that maximizes robustness.

Why Hybrid Systems Win

There are several compelling reasons to design AI systems with deterministic components and fallback paths:

a. Reliability and Safety

AI models can fail in unpredictable ways: edge cases, data drift, adversarial input, or low-confidence predictions. Deterministic logic provides a safety net, ensuring that critical operations continue under known, tested conditions.

b. Explainability and Compliance

Regulated domains - healthcare, finance, law - require transparent decision-making. A hybrid approach allows us to explain final decisions even if an ML model contributed part of the reasoning.

c. Performance and Efficiency

AI is computationally expensive. In many scenarios, we can use simple rules to handle routine cases and reserve AI for ambiguous or high-value decisions. This not only improves performance but also reduces costs.

d. User Trust

Users are more likely to trust a system that admits uncertainty and defers when appropriate. Graceful fallback demonstrates design maturity and builds credibility over time.

The Architecture of Deliberate Hybrid Design

Designing hybrid systems is not an afterthought - it requires planning at the architectural level. A deliberate hybrid system typically includes four layers:

Sensing / Input Layer – Collects data or user input.
AI Layer – Performs probabilistic inference, classification, prediction, or generation.
Deterministic Layer – Enforces rules, policies, and business logic.
Fallback / Escalation Layer – Defines what happens when AI output is unreliable or ambiguous.

Let’s explore each in more detail.

Confidence as a First-Class Citizen

A foundational principle in hybrid design is confidence scoring. Whether you’re dealing with a classifier, a recommender, or a large language model (LLM), you need a way to quantify uncertainty. This could be a probability score, entropy measure, thresholded similarity, or custom metric.
Once you have a confidence signal, you can define thresholds that trigger deterministic behavior. For example:

High confidence: Accept AI output automatically.
Medium confidence: Pass output through a rule-based validation layer.
Low confidence: Trigger fallback (e.g., human review, default rule, or simpler algorithm).

This tiered approach allows systems to dynamically adjust their behavior based on how “sure” they are - a cornerstone of graceful fallback.

Multi-Layer Decision Logic

In many applications, decisions don’t have to be binary (AI vs. rules). Instead, they can flow through multiple layers:

Pre-filter with rules: Before invoking AI, use deterministic logic to discard irrelevant input or enforce hard constraints.
Apply AI model: Perform classification, prediction, or generation.
Validate with rules: Post-process the AI output using deterministic checks.
Fallback or escalate: If validation fails, fallback to a safe default, prompt for human review, or invoke a simpler model.

For example, a medical triage chatbot might:

Validate that symptoms fall within known categories.
Use an AI model to suggest possible causes.
Apply clinical rules to flag life-threatening risks.
Escalate to a human doctor if the risk is high or the model is uncertain.

This layered flow ensures that each decision is made at the right level of complexity and accountability.

Human-in-the-Loop as a Design Pattern

Graceful fallback doesn’t always mean reverting to rules - sometimes, it means involving humans strategically. “Human-in-the-loop” (HITL) systems use people to validate or override AI decisions in critical moments.

For example:

In document review, AI can classify and prioritize, while humans verify.
In autonomous systems, humans can assume control when conditions are unclear.
In support chatbots, unresolved conversations can seamlessly escalate to human agents.

The key is designing handoffs that feel natural - not as emergency patches, but as integral parts of the user experience.

Case Studies: Hybrid Design in Practice

Case Study 1: E-commerce Recommendation Engine

A major retailer built a product recommendation system powered by deep learning. However, they faced two issues: legal restrictions on personalized offers in certain regions and customer complaints about irrelevant suggestions.

The solution was a hybrid pipeline:

Stage 1: Business rules filtered out restricted categories and enforced legal constraints.
Stage 2: The AI model generated ranked recommendations.
Stage 3: A rule-based validator ensured diversity and compliance before display.
Fallback: If the AI model produced low-confidence results, a deterministic “bestsellers” list was shown instead.

Result: higher user satisfaction, full regulatory compliance, and reduced reliance on expensive model inference.

Case Study 2: Industrial Predictive Maintenance

An industrial IoT platform used machine learning to predict equipment failures. But false-positives were costly, and false-negatives were dangerous.

The hybrid solution combined:

Rules: Safety-critical thresholds (e.g., temperature > 120°C) triggered immediate shutdown.
AI: Predictive models forecasted potential failures based on historical data.
Fallback: If predictions were low-confidence or conflicted with sensor data, the system defaulted to conservative rule-based safety actions.

Result: downtime was reduced without compromising safety - and operators trusted the system more.

Patterns for Graceful Fallback

Here are some proven design patterns you can apply in your own systems:

Default Response Pattern: Provide a deterministic “safe answer” if AI is uncertain.
Rule-Gated AI Pattern: Use rules to constrain AI input/output within acceptable boundaries.
Confidence Escalation Pattern: Route low-confidence cases to human review or secondary logic.
Shadow Mode Pattern: Run AI in parallel with existing rule-based systems, comparing outputs before full deployment.
Tiered Complexity Pattern: Start with simple logic; escalate to more complex (AI) methods only when needed.

Important: These patterns are not mutually exclusive - they often work best in combination.

Organizational Mindset: Engineering for Uncertainty

Building hybrid systems is not just a technical challenge - it’s a cultural one. It requires teams to shift their mindset:

From “AI will solve it” TO “AI is a tool, not an oracle”;
From “deterministic vs. probabilistic” TO “deterministic and probabilistic”;
From “one-size-fits-all” TO “context-aware decision flows”. This mindset influences everything: architecture, testing, DevOps, product design, and even how you communicate capabilities to users. It also means measuring success differently. Instead of chasing model accuracy alone, focus on system reliability, user trust, and graceful degradation under uncertainty.

Conclusion: Hybrid Is the Future

The era of AI-only systems is ending. As we integrate machine learning (ML) into mission-critical workflows, robustness matters as much as intelligence. Deliberate hybrid design - the thoughtful fusion of probabilistic models, deterministic logic, and human judgment - is how we get there.
The best systems of the future will not be those that rely on AI blindly. They will be those that understand when to trust the model, when to fall back, and when to escalate. They will embrace uncertainty not as a weakness, but as a design constraint.
And they will succeed because of it.

Key Takeaways:

AI is powerful but inherently uncertain; deterministic logic provides safety, transparency, and reliability.
Design fallback paths intentionally - don’t bolt them on after failures occur.
Use confidence scoring, layered decision flows, and human-in-the-loop mechanisms.
Measure system success in terms of robustness, trust, and graceful degradation - not just model performance.

In software architecture, as in aviation, the most advanced autopilot is only as good as its ability to hand control back to a human pilot. Deliberate hybrid design ensures that when - not if - AI falters, the system continues to serve users reliably and safely.

Key references

“Hierarchical Fallback Architecture for High Risk Online Machine Learning Inference” – Gustavo Polleti et al.
- Proposes a hierarchical fallback architecture for robustness in ML systems; includes fallback modes when inference fails.
- This aligns with the idea of switching from a neural path to fallback logic under failure conditions.
- https://arxiv.org/html/2501.17834v1
“Hybrid AI Reasoning: Integrating Rule-Based Logic with LLMs” (Preprint)
- Explores blending deterministic rule logic with transformer-based models.
- Mentions dual-stream frameworks and fallback or validation layers.
- https://www.preprints.org/manuscript/202504.1453/v1
“Modular Design Patterns for Hybrid Learning and Reasoning Systems”
- Surveys many hybrid architectures, laying out patterns for mixing symbolic/rule and statistical (neural) systems.
- Useful for seeing how fallback or integration can appear in practical systems.
- https://arxiv.org/abs/2102.11965
“Taxonomy of Hybrid Architectures Involving Rule-Based Systems”
- Based on clinical decision systems in healthcare. Shows how rule-based logic is used alongside ML.
- https://pubmed.ncbi.nlm.nih.gov/37355025/
“Hybrid Neuro-Symbolic Learning and Reasoning for Resilient Systems”
- Mentions fallback actions conceived as “Force Minimal Operation” in hybrid systems.
- https://www.sciencedirect.com/science/article/pii/S0960148125020658
“Unlocking the Potential of Generative AI through Neuro-symbolic Architectures”
- Systematic study of architectures that integrate symbolic and neural components.
- https://arxiv.org/html/2502.11269v1

Bibliography

Amershi, S. et al. (2019). Guidelines for Human-AI Interaction. CHI ’19. https://www.microsoft.com/en-us/research/wp-content/uploads/2019/01/Guidelines-for-Human-AI-Interaction-camera-ready.pdf
Varshney, K. R. (2017). Engineering Safety in Machine Learning. 2017 Information Theory and Applications Workshop. https://ieeexplore.ieee.org/document/7888195
Sculley, D. et al. (2015). Hidden Technical Debt in Machine Learning Systems. NeurIPS. https://proceedings.neurips.cc/paper_files/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf
Ribeiro, M. T. et al. (2016). "Why Should I Trust You?": Explaining the Predictions of Any Classifier. KDD. https://arxiv.org/abs/1602.04938
Aston Jonathan (2024). The evolution of hybrid AI: where deterministic and statistical approaches meet. https://www.capgemini.com/be-en/insights/expert-perspectives/the-evolution-of-hybrid-aiwhere-deterministic-and-probabilistic-approaches-meet/
Google Cloud (2023). Machine Learning Design Patterns. (Lakshmanan, Robinson, Munn). O’Reilly. https://www.oreilly.com/library/view/machine-learning-design/9781098115777/
Microsoft (2022). Responsible AI Standard v2. (Sections on fallback and human oversight). https://cdn-dynmedia-1.microsoft.com/is/content/microsoftcorp/microsoft/final/en-us/microsoft-brand/documents/Microsoft-Responsible-AI-Standard-General-Requirements.pdf

Forem: Gelu Vac

Security by Design in the Age of Artificial Intelligence: Fundamentals, Risks and Resilience Strategies

From traditional security to algorithmic security

Vulnerabilities specific to the AI ​​era

Data poisoning

Adversarial Attacks

Model Inversion & Data Extraction

Prompt Injection (in LLMs)

Integrating Security into the AI ​​Lifecycle

Zero Trust and the new paradigm of trust

Regulation, governance and accountability

The ethical dimension of security

Red Teaming and Operational Resilience

Future Challenges

Conclusion: Security as the Foundation of Trust

Bibliography

Deliberate Hybrid Design: Building Systems That Gracefully Fall Back from AI to Deterministic Logic

The Myth of AI Supremacy

Why Hybrid Systems Win

a. Reliability and Safety

b. Explainability and Compliance

c. Performance and Efficiency

d. User Trust

The Architecture of Deliberate Hybrid Design

Confidence as a First-Class Citizen

Multi-Layer Decision Logic

Human-in-the-Loop as a Design Pattern

Case Studies: Hybrid Design in Practice

Case Study 1: E-commerce Recommendation Engine

Case Study 2: Industrial Predictive Maintenance

Patterns for Graceful Fallback

Organizational Mindset: Engineering for Uncertainty

Conclusion: Hybrid Is the Future

Key Takeaways:

Key references

Bibliography

Vulnerabilities specific to the AI era

Integrating Security into the AI Lifecycle