Jefferson's Wheel

Engineering and Technology Magazine (a publication of the British Institution of Engineering and Technology) has an article that highlights adversarial machine learning research: Artificial intelligence: the new ghost in the machine, 10 October 2018, by Chris Edwards.

Although researchers such as David Evans of the University of Virginia see a full explanation being a little way off in the future, the massive number of parameters encoded by DNNs and the avoidance of overtraining due to SGD may have an answer to why the networks can hallucinate images and, as a result, see things that are not there and ignore those that are.
…
He points to work by PhD student Mainuddin Jonas that shows how adversarial examples can push the output away from what we would see as the correct answer. “It could be just one layer [that makes the mistake]. But from our experience it seems more gradual. It seems many of the layers are being exploited, each one just a little bit. The biggest differences may not be apparent until the very last layer.”
…
Researchers such as Evans predict a lengthy arms race in attacks and countermeasures that may on the way reveal a lot more about the nature of machine learning and its relationship with reality.

The New York Times has an article, How Game Apps That Captivate Kids Have Been Collecting Their Data about a lawsuit the state of New Mexico is bringing against app markets (including Google) that allow apps presented as being for children in the Play store to violate COPPA rules and mislead users into tracking children. The lawsuit stems from a study led by Serge Egleman’s group at UC Berkeley that analyzed COPPA violations in children’s apps. Serge was an undergraduate student here (back in the early 2000s) – one of the things he did as a undergraduate was successfully sue a spammer.

The original paper about the study: “Won’t Somebody Think of the Children?” Examining COPPA Compliance at Scale, Irwin Reyes, Primal Wijesekera, Joel Reardon, Amit Elazari Bar On, Abbas Razaghpanah, Narseo Vallina-Rodriguez, and Serge Egelman. Proceedings on Privacy Enhancing Technologies (PETS) 2018.

Serge Egelman, a researcher with the International Computer Science Institute and the University of California, Berkeley, helped lead the study of nearly 6,000 children’s Android apps

Nathaniel Grevatt (“GDPR-Compliant Data Processing: Improving Pseudonymization with Multi-Party Computation”), Matthew Wallace and Parvesh Samayamanthula (“Deceiving Privacy Policy Classifiers with Adversarial Examples”), and Guy Verrier (“How is GDPR Affecting Privacy Policies?”, joint with Haonan Chen and Yuan Tian) presented posters at USENIX Security Symposium 2018 in Baltimore, Maryland.

There were also a surprising number of appearances by an unidentified unicorn:

Your poster may have made the cut for the #usesec18 Poster Reception, but has it received the approval of a tiny, adorable unicorn? @UVA #seenatusesec18 #girlswhocode #futurecomputerscientist #dreambig pic.twitter.com/bZOO6lYLXK

— USENIX Security (@USENIXSecurity) August 16, 2018

I gave a keynote talk at USENIX Workshop of Offensive Technologies, Baltimore, Maryland, 13 August 2018.

The title and abstract are what I provided for the WOOT program, but unfortunately (or maybe fortunately for humanity!) I wasn’t able to actually figure out a talk to match the title and abstract I provided.

The history of security includes a long series of arms races, where a new technology emerges and is subsequently developed and exploited by both defenders and attackers. Over the past few years, “Artificial Intelligence” has re-emerged as a potentially transformative technology, and deep learning in particular has produced a barrage of amazing results. We are in the very early stages of understanding the potential of this technology in security, but more worryingly, seeing how it may be exploited by malicious individuals and powerful organizations. In this talk, I’ll look at what lessons might be learned from previous security arms races, consider how asymmetries in AI may be exploited by attackers and defenders, touch on some recent work in adversarial machine learning, and hopefully help progress-loving Luddites figure out how to survive in a world overrun by AI doppelgängers, GAN gangs, and gibbon-impersonating pandas.

I co-organized, with Homa Alemzadeh and Karthik Pattabiraman, a workshop on trustworthy machine learning attached to DSN 2018, in Luxembourg: DSML: Dependable and Secure Machine Learning.

Group Dinner

Including our newest faculty member, Yongwhi Kwon, joining UVA in Fall 2018!

Yuan Tian, Fnu Suya, Mainuddin Jonas, Yongwhi Kwon, David Evans, Weihang Wang, Aihua Chen, Weilin Xu

Poster Session

Fnu Suya (with Yuan Tian and David Evans), Adversaries Don’t Care About Averages: Batch Attacks on Black-Box Classifiers [PDF]	Mainuddin Jonas (with David Evans), Enhancing Adversarial Example Defenses Using Internal Layers [PDF]

I spoke on Lessons from the Last 3000 Years of Adversarial Examples at Huawei’s Strategy and Technology Workshop in Shenzhen, China, 15 May 2018.

We also got to tour Huawei’s new research and development campus, under construction about 40 minutes from Shenzhen. It is pretty close to Disneyland, with its own railroad and villages themed after different European cities (Paris, Bologna, etc.).

Huawei’s New Research and Development Campus [More Pictures]

Unfortunately, pictures were not allowed on our tour of the production line. Not so surprising that nearly all of the work was done by machines, but was surprising to me how much of the human work left is completely robotic. The human workers (called “operators”) are mostly scanning QR codes on parts, and following the directions that light up with they do, or scanning bins and following directions on a screen to collect parts from bins and scanning them when they are put into the bin. This is the kind of system that leads to remarkably high production quality. The parts are mostly delivered on tapes that are fed into the machines, and many machines along the line are primarily for testing. There is a “bottleneck” marker that is placed on any points that are holding up the production line.

The public (at least to the factory) “grapey board” keeps track of the happiness of the workers — each operator puts up a smiley (or frowny) face on the board to show their mood for the day, monitored carefully by the managers. There is a batch of grapes to show performance for the month. If an operator does something good, a grape is colored green; if they do something bad, a grape is colored black. There was quite a bit of discussion among the people on the tour (mostly US and European-based professors) if such a management approach would be a good idea for our research groups… (or for department chairs for their faculty!)

In front of Huawei’s “White House”, with Battista Biggio [More Pictures]

Weilin Xu presented Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks at the Network and Distributed System Security Symposium 2018. San Diego, CA. 21 February 2018.

Paper: Weilin Xu, David Evans, Yanjun Qi. Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks. NDSS 2018. [PDF]

Project Site

The 24^th ACM Conference on Computer and Communications Security was held in Dallas, 30 October – 3 November. Being Program Committee co-chair for a conference like this is a full-year commitment, and the work continues throughout much of the year preceding the conference. The conference has over 1000 registered attendees, a record for any academic security research conference.

Here are a few highlights from the conference week.

PC Chairs’ Welcome (opening session)

Giving the PC Chairs’ Welcome Talk

Audience at Opening Session

ACM CCS 2017 Paper Awards Finalists

CCS 2017 Awards Banquet

At the Award’s Banquet, I got to award a Best Paper award to SRG alum Jack Doerner (I was, of course, recused by conflict from being involved in any decisions on his paper).

UVA Lunch (around the table starting at front left): Suman Jana (honorary Wahoo by marriage), Darion Cassel (SRG BSCS 2017, now at CMU), Will Hawkins, Jason Hiser, Samee Zahur (SRG PhD 2016, now at Google), Jack Doerner (SRG BACS 2016, now at Northeastern), Joe Calandrino (now at FTC); Back right to front: Ben Kreuter (now at Google), Anh Nguyen-Tuong, Jack Davidson, Yuan Tian, Yuchen Zhou (SRG PhD 2015, now at Palo Alto Networks), David Evans.

I gave a talk at the First ACM Workshop for Women in Cybersecurity (affiliated with ACM CCS 2017) on Truth, Social Justice (and the American Way?):

There’s also a short paper, loosely related to the talk: [PDF]