Page 14 of 14 · Neural networks & security
ML security and ethics: when the model is the target
Every page so far assumed the data was honest. Attackers do not cooperate. A machine learning system has every vulnerability of ordinary software, plus new ones: its behavior comes from data, so whoever influences the data or the inputs can influence the decisions.
The ML attack surface
Select a stage of the pipeline from page 02 to see how it can be attacked.
Attack a malware detector
The detector below is a logistic regression (page 08) trained on two features of executable files. Try two attacks against it.
A field guide to attacks on ML
| Attack | What the attacker does | Main defenses |
|---|---|---|
| evasion (adversarial examples) | Crafts inputs that are misclassified at prediction time, such as malware padded to look benign or a stop sign with stickers. | Adversarial training, robust features the attacker cannot cheaply change, ensembles |
| poisoning | Slips bad examples into the training data, for example through user-submitted samples or a scraped dataset. | Data provenance, outlier filtering, limiting who can contribute labels |
| backdoor | Poisons training so the model behaves normally except when a secret trigger appears. | Trusted training pipelines, model scanning, verifying third-party models |
| model extraction | Queries a model's API many times and trains a copy from the answers, stealing the work and making evasion easier. | Rate limiting, query monitoring, returning labels instead of exact probabilities |
| membership inference | Determines whether a specific person's record was in the training data, a privacy leak. | Regularization (page 07), differential privacy, less confident outputs |
| prompt injection | Hides instructions in text an LLM reads, such as a web page or email, to hijack what it does. | Treat model output as untrusted, least-privilege tool access, human approval for sensitive actions |
| supply chain | Tampers with libraries, pretrained models, or model files. Some model file formats, such as Python pickle files, can run code when loaded. | Pin dependencies, verify hashes, prefer safe formats such as safetensors |
Name that attack
Ethics: harm without an attacker
Some of the most serious ML failures involve no attacker at all. A model learns whatever patterns are in its data, including unfair ones.
A model trained on past hiring decisions can learn past discrimination. Face recognition trained mostly on one group performs worse on others. Measure performance separately for each group, not just overall.
Models can memorize and leak training data, including personal information. Collect only what you need and control who can query the model.
People affected by automated decisions, such as being flagged for fraud, deserve an explanation and a way to appeal to a human.
"The model decided" is never an excuse. A person or organization owns every deployed model and its mistakes.
- ML adds attack surface at every pipeline stage: data, training, model files, the prediction API, and inputs.
- Evasion attacks fool a trained model; poisoning corrupts it during training; extraction and membership inference steal from it.
- Responsible ML also means checking for bias, protecting privacy, explaining decisions, and keeping humans accountable.