Attacks on Real-World Uses of Machine Learning


Machine-learning (ML) classifiers have numerous real-world applications, some of which are security-critical.  Motivated by such applications, much research has been devoted to better understanding adversarial examples, which are specially crafted inputs to a machine-learning model that are perceptually similar to benign inputs but are classified differently (i.e., misclassified). Both algorithms that create adversarial examples and strategies for defending against adversarial examples typically use Lp-norms (usually, p = 0, 2, or ∞) to measure the perceptual similarity between an adversarial input and its benign original.  That is, attacks can leverage any perturbation of sufficiently small Lp-norm to modify benign inputs so as to change their classifications.

Unfortunately, relying on Lp-norm bounds to enforce that adversarial examples are perceptually similar to the benign inputs from which they are derived is problematic, for several reasons.  First, we have shown that a small Lp-norm is neither sufficient nor necessary for a human to classify an adversarial example the same as the benign input from which it was derived.  That is, small Lp-norms are poor surrogates for perceptual similarity.  Second, in many real-world scenarios, the constraints that dictate how an adversary can manipulate a benign input cannot be captured simply by bounded Lp-norms.  For example, in face-recognition applications, an adversary might be able to modify his own appearance only through wearing accessories (e.g., eyeglasses) that are inconspicuous to an onlooker.  As another example, if the classifier is used to determine whether an executable is malware, the adversary might be able to modify the malware arbitrarily as long as its functionality is not changed.

Project Description

This project is focusing on how to create and defend against adversarial examples in realistic applications of ML classifiers.  To date we have focused on two scenarios, namely the face-recognition and malware-detection scenarios mentioned above.

In the context of face recognition, we have developed methods by which an attacker can produce eyeglass frames that, when worn, cause the wearer to be misclassified.  Our attacks are physically realizable and inconspicuous, and allow an attacker to evade recognition or impersonate another individual.  More specifically, our methods systematically compute the color and pattern for eyeglass frames, which can then be printed on a commodity 2D or 3D printer.  When worn by the attacker whose image is supplied to a state-of-the-art face-recognition algorithm, the eyeglasses allow her to evade being recognized or to impersonate another individual.

Regarding malware detection, both researchers and anti-virus vendors have proposed neural networks for malware detection from raw bytes that do not require manual feature engineering.  We have developed an attack that interweaves binary-diversification techniques and optimization frameworks to mislead such classifiers while preserving the functionality of the binaries.  Unlike prior attacks, ours manipulates instructions that are a functional part of the binary, which makes it particularly challenging to defend against. We evaluated our attack against three neural networks, finding that it often achieved success rates near 100%. Moreover, we found that our attack can fool some commercial anti-viruses, in certain cases with a success rate of 85%.


M. Sharif, S. Bhagavatula, L. Bauer, and M. K. Reiter. Accessorize to a crime: Real and stealthy attacks on state-of-the-art face recognition. In 23rd ACM Conference on Computer and Communications Security, pages 1528-1540, October 2016.

M. Sharif, L. Bauer, and M. K. Reiter. On the suitability of Lp-norms for creating and preventing adversarial examples. In 2018 Workshop on The Bright and Dark Sides of Computer Vision: Challenges and Opportunities for Privacy and Security, pages 1718-1726, June 2018.

M. Sharif, S. Bhagavatula, L. Bauer, and M. K. Reiter. A general framework for adversarial examples with objectives. ACM Transactions on Privacy and Security 22(3), June 2019.

K. Lucas, M. Sharif, L. Bauer, M. K. Reiter, and S. Shintre. Malware makeover: Breaking ML-based static analysis by modifying executable bytes. In 16th ACM Asia Conference on Computer and Communications Security, June 2021.

2016 - present
For More Information

Please contact Prof. Reiter for further information.