On Message with Ben Gross

The Electric Alchemy security consulting firm has an interesting post about Cracking Passwords in the Cloud: Breaking PGP on EC2 with EDPR describing their experiences running Elcomsoft’s Distributed Password Recovery tool across 100 Amazon Elastic Computing Cloud (EC2) instances. The goal was to crack the passwords of several files in the old PGP ZIP format. The results clearly show that the cost of recovering relatively simple passwords that are all lower case and less than 10 letters, has become affordable. Longer and more complex passwords with mixed case and symbols are still prohibitive to recover. The graphs make it easy to see that small amounts of additional complexity dramatically increase the difficulty of recovering the password. Electric Alchemy posted a follow up as Cracking Passwords in the Cloud Q&A.

In his post, Twitter, OAuth and Passwords – Oh My!, Terence Eden describes a potential risk with OAuth if a user changes their password on a service such as Twitter, but still has active OAuth tokens that are not invalidated. He proposes a scenario where a user believes he has lost control of his password and changes it, but does not realize that an attacker has also authorized other services to access the Twitter account and still has access. The OAuth developers point out that this is not a design flaw as typically users would not want to invalidate all of their OAuth tokens, but that they should check to make sure if they have personally authorized all the services linked to that OAuth site. The resulting discussion in comments and related posts is interesting. I do believe that when a user changes a password on a site, the service should all of their active OAuth tokens for inspection along with an interface to invalidate them at the same time in case the attacker created new OAuth-based authorizations. Eden’s points out a more complex problem where the attacker might have authorized an OAuth token for a service that the user has already previously authorized.

The article 30 years of failure: the username/password combination by John Timmer in Ars Technica describes a recent study published in the 2009 Proceedings of the Human Factors and Ergonomics Society Hoonakker et. al. The study included two rounds of focus groups and a survey given to the employees of a large organization that asked about password habits. They received 836 responses that illustrated a litany of problems users have associated with remembering and managing passwords. The results are not surprising, but detailed breakdown in the analysis is valuable for determining the scope of the problem. The authors indicate the tradeoffs between security and usability are significant and that we need to find ways to make it easier for users to maintain good security practices. The paper includes a nice overview of previous research and current options for improving the situation.

What we think is reasonable, commonplace, or even possible in terms of protecting or violating online privacy shifts constantly. Recent developments in tools and techniques for tracking online behavior and identifying individuals from supposedly anonymized data sets should cause us to reevaluate what is possible.

Katherine McKinley of iSEC Partners published a detailed analysis of how popular browsers and browser extensions handle cookies and other methods of local data storage used for tracking users in her December, 2008, paper Cleaning Up after Cookies (PDF). McKinley tested the ability for browsers and extensions to clear the private data as well as “private browsing” features. She found that most browsers attempted to clear previous stored private data, but often left some data accessible. She found that Adobe Flash did not attempt to remove this data and in fact stored it in such a way that it circumvented most privacy protections offered by browsers. iSEC Partners created an online version of the test used in the article to allow individuals to test their own configurations. It is available at Breadcrumbs Tracker.

The August, 2009 paper Flash Cookies and Privacy by Ashkan Soltani and Shannon Canty and Quentin Mayo and Lauren Thomas and Chris Jay Hoofnagle at UC Berkeley focuses directly on the privacy issues related to Flash Cookies. The authors survey the top 100 web sites according to QuantCast in July of 2009 and found that more than half of them used Flash cookies. The authors note that unlike standard HTTP cookies, Flash cookies do not have an expiration date and are stored in a different location on the file system that is harder to find. Most cookie management tools will not delete these type of cookies and they remain in place even when private browsing mode in enabled. The authors found that Flash cookies were frequently employed to track users that had explicitly attempted to prevent cookie tracking by using the Flash cookie to regenerate a HTTP cookie that had been deleted.

Most online services use multiple tracking services for analytics, performance monitoring, and usability analysis. A mixture of JavaScript-based tracking codes and cookies is the most common method for user tracking. The paper On the Leakage of Personally Identifiable Information Via Online Social Networks (PDF) presented at the ACM Workshop on Online Social Networks by Balachander Krishnamurthy and Craig Wills describes the techniques used by advertising firms and social networks services to track users and the types of information they release. The authors studied information leakage from twelve online social networks and found that the bulk of user information is released through HTTP headers and third-party cookies.

In his post Netflix’s Impending (But Still Avoidable) Multi-Million Dollar Privacy Blunder on the Freedom to Tinker blog, Paul Ohm discusses his 2009 publication Broken Promises of Privacy: Responding to the Surprising Failure of Anonymization in the context of the announcement for the second Netflix prize to improve the accuracy of Netflix predictions. Ohm argues that it is not possible to anonymize the data and that it is irresponsible and possibly illegal to release it. Netflix released a half a million anonymized subscriber records for analysis in the original contest. The one million dollar prize resulted in significant numbers of researchers entering the contest.

Soon after the Netflix released the records, researchers Arvind Narayanan and Vitaly Shmatikov proved they could identify individual subscribers by combining the Netflix data with other databases. Their publication Robust De-anonymization of Large Sparse Datasets (PDF) presented at the 2008 IEEE Symposium on Security and Privacy describes How to Break Anonymity of the Netflix Prize Dataset. Narayanan and Shmatikov continued their research and demonstrated de-anonymizing social networks such as Twitter in De-Anonymizing Social Networks (PDF) (paper FAQ) a paper presented at the 2009 IEEE Symposium on Security and Privacy. Ohm, reminds the readers about the scandal that occurred in 2006 when AOL researchers Greg Pass, Abdur Chowdhury, and Cayley Torgeson presented their paper A Picture of Search (PDF) at the first International Conference on Scalable Information Systems. The authors released an anonymized dataset they analyzed in the paper that included more than six hundred thousand AOL users, some individuals were subsequently individually identified.

Carnegie Mellon University professor Latanya Sweeney developed the foundation for much of the current work on de-anonymizing data sets. Her paper All the Data on All The People (editor: abstract is no longer publicly available; contact author) published in 2000, showed that it was possible to identify individuals in US Census data using only a few variables. The paper argues that it is possible to identify almost 90% of the US population using only full date of birth, gender, and ZIP code.

Alessandro Acquisti and Ralph Gross (no relation) presented their research on Predicting Social Security Numbers from Public Data. The authors demonstrate that it is possible to automate the process of predicting an individual’s Social Security Number (SSN) for large portions of the population using public information. The information used to create predictions is easily harvested from social networking sites, voter registration records, and commercial databases. Aquiusti and Gross argue that we must reconsider our policies around the use of SSNs, which are commonly used for authentication and frequently abused by identity thieves.

You should follow me on Twitter.

Campaign Monitor recently updated and reorganized its excellent Guide to CSS support in email clients. I find the logical groupings of the new guide make it even more useful. CSS support varies widely across mail clients, which makes Campaign Monitor’s guide is well worth examining for anyone designing HTML email messages that include CSS. A version of the guide with the ten most popular email clients is available in HTML, while a full version of the guide that includes 23 mail clients is available in PDF and Excel formats. Recent updates to the guide include MySpace Mail and seven mobile email clients. The mobile email clients roughly in order of CSS functionality are iPhone 3.0, Android, Android Gmail application, Palm Pre (WebOS), Palm Treo (Windows Mobile 6.5), Palm Treo (Palm Garnet OS), and the Blackberry.

In a post on Smarterware, Gina Trapani makes a good point that you should Never Use Inactive Webmail as Your Secondary Email Account. Trapini says that people often use a infrequently accessed webmail account or other secondary account when signing up with services out of fear that providing their email address might result in an increase in spam. The problem is that most free webmail services deactivate accounts that have not been used in a specified amount of time. Deactivation typically means that the service will begin bouncing mail to that account. After an additional amount of time the account may be deleted entirely and in some cases reassigned to someone else. This can have severe consequences as it did in the Twitter hack story since a deleted hotmail account was assigned as the recovery email address for the Twitter employees Gmail account. Thus the attacker simply registered the recycled Hotmail username and reset the Gmail account. The moral of the story is that you should only register accounts with email addresses you will be able to maintain control over time. If you do register an account with an infrequently used webmail account, put an entry on your calendar to log into the account every few months. People do regularly change primary email addresses and it is not a bad idea to periodically review the email addresses listed in your account information for important services.

Symantec’s latest Spam and Phishing Landscape: September 2009 report finds that while the overall volume of spam decreased two percent since the previous report, they still place the overall spam rate at 87%. Phishing attacks decreased by 45%. There was a 74% decrease in the use of phishing toolkits to automate the phishing process. Symantec hypothesizes this was largely due to the discontinuation of a specific toolkit that targeted a social network site. The discontinuation was likely caused by the shutdown of command and control servers for the toolkit.

Validating data in web forms reduces the likelihood of inadvertent submission of data that is incorrectly formatted, inconsistent, or incomplete. It is often useful to validate email addresses, especially if the addresses are going to be used for receipts or other types of follow up. Validation (and basic bounds checking) can also reduce the chance that email address field could be used as an attack vector.

It is important to note that email addresses can be significantly more complicated than commonly thought. This means that it is important to consult the most current RFCs for email standards and ICANN announcements for new types of Top Level Domain names otherwise valid email addresses may be blocked. For example, the plus character is a valid within the local portion of an email address. The plus is typically used as an optional feature for sub-addressing and is supported in many mail servers, Cyrus IMAP installations, and in Gmail. However, the plus sign is frequently rejected as invalid by many web forms.

Unless there is a specific need for sophisticated email address validation, I recommend that sites limited themselves to very basic validation such as simply checking for an @ sign and possibly characters to either side of it. When sophisticated validation is used, it is important to test the algorithm and make sure it is kept up to date. This Stack Overflow thread, How far should one take e-mail address validation?, details many of the problems with being too clever when validating addresses. There will always be users who purposefully submit incorrect data and while this can be limited somewhat by validation, simply sending a verification email is a far more effective method.

Dave Child’s early posts from 2004, Email Address Validation and Email Address Validation Updated, laid out many of the complexities of more sophisticated email address validation. The comments to the posts brought up edge cases where the script resulted in both false positives and false negatives. Child has continued to revise the script and it is available as a Google Code project php-email-address-validation.

Douglas Lovell’s 2007 Linux Journal article Validate an E-Mail Address with PHP, the Right Way attempted to present and even more complex email validation algorithm along with detailed notes on the requirements relating to the various updated RFCs. The comments to this article also bring up many edge cases, which demonstrate the complexity of accurately validating email addresses. Jochen Topf’s articles, the Anatomy of a Mail Address and Characters in the local part of a mail address, are good introductions to the problem as well.

Dominic Sayers wrote a series of posts that iterated on a further refined algorithm that resulted in the RFC-compliant email address validator. Sayers also produced a set of unit tests with a large collection of email addresses in order to compare his own algorithm against others. His PHP code is regularly updated and is also available on Google Code. Cal Henderson (formerly of Flickr) wrote his own RFC (2)822 & 3696 Email Address Parser in PHP, which also passes 100% of Sayers Unit tests.

The examples above are all in PHP. Unfortunately, I could not find a client-side only validation option in JavaScript that was anywhere near as complete as an of the PHP examples. Hopefully, someone will write one or port one of the PHP versions to JavaScript. Les Hazlewood released a Java-based application for Email Validation using Regular Expressions (the Right Way) and Casey Connor of Boxbe updated Hazlewood’s EmailAddress.java code.

The chapter on inline validation from Luke Wroblewski’s excellent book Web Form Design: Filling in the Blanks describes how inline validation can improve the usability of web forms. He suggests that users should receive immediate feedback on whether or not a given input will be accepted as well as suggestions for correcting invalid input. His blog post Web Form Design: Boingo shows a real world example where inline validation would improve the user experience for a registration form. A recent report Web forms design guidelines: an eyetracking study from cxpartners’ Chui Chui Tan provides even more suggestions on how to best handle inline validation.

In this article, I primarily discuss server-side validation, rather than validation by SMTP commands such as looking for 250 and 550 SMTP response codes as presented in How to check if an email address exists without sending an email?. If the email address is to be used in a mailing list I recommend that systems send an email with a URL that must be clicked for verification so that the address qualifies as double opt-in for compliance with CAN SPAM and most major Email Service Provider requirements.