Hashing and Data Integrity

You have already met hashes several times without a proper introduction: the checksum of a downloaded image, the password hashes in /etc/shadow, the idea that a file was "not modified". Hashing is the simplest and most widely used tool of cryptography, and it answers one question: is this data exactly the same as before? In this lesson you learn what a hash function is, what it guarantees, and what it does not.

The first three chapters of this course gave you a lab, and the next two gave you the systems that run in it. Chapter 6 steps back to the mathematics that secure all of them. We begin with the least intuitive but most useful notion: the fingerprint of a piece of data.

What is a hash function?

A cryptographic hash function takes any input (a word, a file, a whole disk image) and produces a fixed-size output called a hash, digest or fingerprint. A common function, SHA-256, always produces 256 bits, typically written as 64 hexadecimal characters, whether the input is one letter or one terabyte.

Try it yourself on a Linux or macOS terminal (on Windows, Get-FileHash does the same in PowerShell):

printf 'hello' | sha256sum
2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824  -

printf 'Hello' | sha256sum
185f8db32271fe25f561a6fc938b2e264306ec304eda518007d1764826381969  -

On macOS, use shasum -a 256 instead of sha256sum. Look at the two results: changing a single letter, from h to H, produced a completely different hash, with no visible resemblance. This is called the avalanche effect.

An analogy: the hash is like a fingerprint of a document. A fingerprint is small and identifies a person, yet you cannot rebuild the person from the fingerprint. Similarly, you cannot rebuild the file from its hash.

The properties that make a hash useful

• Deterministic: the same input always gives the same output.
• Fast to compute for any input size (for general-purpose hashes).
• One-way (preimage resistance): given a hash, it is infeasible to find an input that produces it, other than by guessing.
• Collision resistance: it is infeasible to find two different inputs with the same hash. Collisions must exist mathematically (infinite inputs, finite outputs), but a good function makes finding one practically impossible.
• Avalanche effect: a tiny change in the input changes the output completely.

It is crucial to understand that a hash is not encryption. Encryption is reversible with the right key; hashing is not designed to be reversed at all. People sometimes say "the password is encrypted" when they mean "hashed", and the confusion leads to design mistakes.

Which functions to use, and which to avoid

Hash functions age as researchers find weaknesses. Some practical guidance, without version-specific details that change over time:

• MD5 and SHA-1 are considered broken for security purposes, because practical collision attacks exist. They may still appear in old systems, and they are sometimes used for non-security purposes such as spotting accidental corruption, but they must not be used where an attacker could craft files.
• The SHA-2 family (SHA-256, SHA-512) and SHA-3 are the recommended general-purpose choices today. Follow the guidance of official bodies such as NIST and, in France, the ANSSI at cyber.gouv.fr, which publish and update recommendations on cryptographic mechanisms.

Always check current recommendations rather than trusting a fixed list, including this one.

Use 1: verifying file integrity

When you download a lab image, the project publishes its checksum. After downloading, you compute the hash of your copy, and if it matches, the file is complete and identical to what the project published. This protects against transmission errors and, when the checksum is obtained through a trusted channel, against tampering.

sha256sum kali-image.iso
# compare with the value published on the official site

Notice the condition: the checksum must come from a trustworthy place, separate from the file itself. If an attacker can modify both the file and its published hash on the same compromised page, the check proves nothing. This is why important projects also sign their checksum files with a digital signature, a mechanism we explore in lesson 3.

In incident response, hashes help too: defenders compute the hash of a suspicious file and look it up in databases of known malware, without sharing the file itself. Forensic investigators hash a disk image before and after analysis to prove that they did not alter it.

Use 2: storing passwords

A website must check your password, but it should never store it in clear text. The classic approach is to store the hash of the password. At login, the server hashes what you typed and compares it to the stored hash. A database leak then reveals hashes, not passwords.

But there is a catch, and it is where beginners go wrong. A general-purpose hash such as SHA-256 is too fast. An attacker who steals a list of hashes can compute billions of guesses per second on suitable hardware, comparing each guess with the stolen hashes. Two further problems arise:

• Identical passwords give identical hashes. If two users choose the same password, the leak shows it, and one guess works for both. Attackers also use precomputed tables of hashes of common passwords.
• Weak passwords fall quickly to guessing, whatever the hash.

The solution has two ingredients:

1. A salt: a random value, unique to each password, mixed with the password before hashing and stored next to the hash. The salt is not secret; its purpose is to make every hash different, so that precomputed tables are useless and identical passwords produce different results.
2. A deliberately slow, specialised password hashing function, such as Argon2, scrypt or bcrypt (or PBKDF2 in some contexts). These functions have a tunable cost that makes each guess expensive, which barely affects a legitimate login but greatly slows down an attacker who must try many guesses.

Look at the /etc/shadow file of your lab server (with sudo). Each line shows a user name and a field that starts with something like $y$ or $6$. These prefixes identify the algorithm, followed by the salt and then the hash. Systems store all of it in one string. In the next courses, you will see how auditors check that password hashes are strong, always with authorisation and in a lab.

The defender's rule of thumb: never invent your own scheme. Use a well-tested library with a recommended password hashing function, with proper parameters.

Use 3: message authentication (HMAC)

A plain hash can be recomputed by anyone, so on its own it does not prove who made it. An HMAC (hash-based message authentication code) mixes a secret key into the hash. Only someone who holds the key can produce or verify the correct value. HMAC is used to protect API requests, cookies and tokens against tampering. When you see a signed cookie or a token that the server can trust without storing it, an HMAC (or a digital signature) is often behind it.

Common pitfalls

• Using fast hashes (MD5, plain SHA-256) alone for passwords.
• Hashing without salt, or with one global salt for everyone.
• Treating a hash as a way of hiding data that has little variety. A hash of a phone number or a date of birth can be reversed by trying every possibility, since there are only so many.
• Comparing hashes obtained from the same untrusted source as the file.
• Thinking that hashing is encryption, or that a hash can be "decrypted".

Your turn: see the avalanche and detect tampering

In your lab, create a text file ~/lab/report.txt with a short sentence. Compute its SHA-256 hash and write it in your notebook. Then change one character in the file and compute the hash again. Next, make a copy of the original file under another name and compare its hash with the original. Finally, answer in your notebook: in which of these cases does the hash prove that the file is unmodified, and why must the reference hash be stored somewhere an attacker cannot change?

echo "Lab report v1" > ~/lab/report.txt
sha256sum ~/lab/report.txt
cp ~/lab/report.txt ~/lab/report-copy.txt
sha256sum ~/lab/report-copy.txt
echo "Lab report v2" > ~/lab/report.txt
sha256sum ~/lab/report.txt