The password strength illusion

Earlier this year we saw massive password databases released onto the dark web. These databases were more extensive than any of the databases leaked before. In January a password database called Collection#1 was released containing millions of passwords. A few weeks later, an even bigger collection was leaked onto the dark web. The collection consisted of several volumes, collectively known as Collection#2-Collection#5.

Finding the collections at the time of the leak was challenging. The team at Crimson Wall managed to locate and download the collections (via torrents). We waited patiently for weeks for the torrent downloads to complete. Finally, after the downloads completed, we could inspect all the password collections (Collection#1-Collection#5).

Initial inspection of the collections revealed that the dumps were as bad as everyone feared. The dumps contained millions of plaintext passwords and password hashes (that anyone can de-hash using hashcat). These dumps seem to have been obtained by hacking websites and then dumping the user credentials. The collections each consist of thousands of files. The files and directories are named appropriately to indicate the origin of the dumped credentials. The dumps also contained usernames and password stolen from various South African websites. To protect the identity of the compromised websites, we’ll only show a rough calculation performed to count the amount of .co.za websites in the dumps.

The estimate indicates that roughly 73 .co.za domains were compromised. During investigations, we found that some dumps files were duplicated among the collections. For this reason, we’ll call this number an “estimate” – please don’t quote us on this.

Interesting collection facts

The password collections consist of thousands of text files. Some text files contain plaintext passwords, while others contain hashed passwords. In some text files it’s obvious if the password is hashed or not, but in others, it’s not the case. We created a python script that used hashid to find and extract passwords that are in plaintext format.

All passwords identified to be in plaintext format were added to a list. The list was sorted according to popularity, thereby giving us a list of the most common (plaintext/de-hashed) passwords found in the password dumps. The table below shows the 1000 most popular plaintext passwords found in all the password collections:

As can be seen from the table above, the top passwords are horrific. Passwords further down the list seem to be personal in nature as passwords tend to be the names of family members, and other meaningful names. What is also interesting is that many of the passwords, (especially the passwords further down the list) conform to password composition rules enforced by many organizations.

Strong password policies

As seen in the top 1000 password list, there are many guessable passwords that can easily be deduced from a person’s social media profile (like the name of a child or pet). To overcome the problem of easily guessable passwords, organizations enforce password policies. The screenshot below shows an actual password policy of a well-known online system. We have all seen policies like these – some policies are more permissive while and others are more restrictive.

Having a password policy will, at the very least, force users not to use an exceptionally poor password. However, users adapt as password policies adapt. In organisations with strong password policies it’s not uncommon for users to choose a password that conforms to the password policy but is still easy to crack. For example, the password September2019! conforms to the password policy shown above, yet the password follows a predictable pattern that is (unsurprisingly) popular in organisations where password composition policies are enforced, making it a poor password.

A more interesting side-effect of password policies is that strict password policies may reduce the brute-force search space, should an attacker choose to try every possible password combination. For example, we know that all individuals with password conforming to the password policy above will have passwords that are between 8 and 16 characters. We know that each password will contain at least one number and one special character.

Assuming that the password composition policy discussed above is used, an attacker performing a brute-force attack to uncover passwords does not have to evaluate 123456789012345, abcdefghijklmno, or !@#$%^&*()!@#$%^ as these combinations do not conform to the password composition policy. However, if the password composition policy did not exist, then these combinations need to be evaluated as well. Theoretically speaking a password policy can help to reduce the size of the password search space, thereby making it possible (in theory) for the attacker to find the password, using a brute-force attack, with less effort.

Before we go any further, we need to discuss how such an attack would work. Brute-forcing an actual login service is unlikely to yield any meaningful results as accounts are typically locked after a few unsuccessful login attempts. Knowledge of the password policy becomes useful after a system compromise when a password database has been dumped. Passwords are typically salted and hashed when stored in a database. This means that passwords are stored in a way which makes it very difficult to determine what the actual passwords are.

When a hashed password database is stolen, it is possible to recover passwords using a process called de-hashing. De-hashing is typically performed by calculating the hashes for brute-force password combinations and comparing the hashes produced by the process to a hashes stored in the database. This is a time-consuming process (especially for salted hashes) as all possible password combinations will in concept have to be tested unless some information about the stored password is available to reduce the size of the password search space. As luck might have it, a password policy assists an attacker by reducing the brute-force password search space size, thereby allowing an attacker to de-hash password quicker.

How do we solve this problem?

Think about the problem – a hashed password (under normal conditions) has a fixed length. Increasing the length of a password has no impact on the storage requirement of the password hash as the hash length is concept always the same (ignoring salt, encoding, and padding issues). The question that we all need to ask ourselves is if enforcing an upper-bound on a password length is sensible.

For example, a password like “I love computers, but I hate stinky cheese.” is a better password than “September2019!”, yet it does not conform to the password policy discussed earlier as it is more than 16 characters in length and it does not contain a number.

The longer password contains no forced characters, no forced symbols, and it’s easy to remember. The brute-force search space increased drastically since the password is very long (and assumptions cannot be made about the content of the password). In short – an attacker wastes a lot of time and effort to de-hash the password.

In the NIST Special Publication 800-63B, NIST discussed password policy recommendations. What is interesting about the recommendations is that the recommendations are different from what most organizations are doing at this moment in time.

The most important parts are:

• Password needs to be at least 8 characters in length up to at least 64 characters.
• Password composition rules should not be enforced.
• Password changes should be enforced when it is evident that passwords were compromised.
• Passwords should be checked against dictionaries and passwords found in previous password breaches. Passwords found in these dictionaries should not be allowed.
• Password strength meters should guide users when creating passwords.

The specification also discusses many other details as well, so be sure to read it. The most interesting thing about password composition rules is that almost all organizations are using them as they are widely believed to work. However, as discussed in this article, a good password is not necessarily a password that conforms to strict composition rules.

Breached password database

Crimson Wall’s vision and mission statement show our commitment to improving the state of information security for everyone.

We created a dataset containing the 10,000,000 most popular passwords found in the password collections (sorted by popularity). We are making the dataset freely available for download. The dataset can be used, among others things, to perform breached password validations.

The dataset is updated from time-to-time as we find more de-hashed password databases, so be sure to check for updates.