An investigator with a disk image of a thousand files does not read them one at a time, hoping to notice something. The very first script run against new evidence is almost always one that catalogues it: what exists, what type each file really is, and a hash proving the state each was found in.
Analogy: The library catalogue
You cannot find a book in a library with no catalogue; you would walk every shelf. Before an investigator can search a disk image for anything meaningful, they build a catalogue of it: what files exist, their type, their size, and a hash to prove the state each was found in.
Building the index
build_index.py
# build_index.py: walk a directory and catalogue every file
import os
import hashlib
import json
import magic
from forensic_log import log_action
def hash_file(path, algorithm="sha256", chunk_size=65536):
hasher = hashlib.new(algorithm)
with open(path, "rb") as f:
while chunk := f.read(chunk_size):
hasher.update(chunk)
return hasher.hexdigest()
def build_index(root_dir, logfile="forensic_run.log"):
index = []
detector = magic.Magic(mime=True)
for dirpath, _, filenames in os.walk(root_dir):
for name in filenames:
path = os.path.join(dirpath, name)
try:
record = {
"path": path,
"size_bytes": os.path.getsize(path),
"mime_type": detector.from_file(path),
"sha256": hash_file(path),
}
index.append(record)
log_action(logfile, f"Indexed {path}")
except (PermissionError, OSError) as exc:
log_action(logfile, f"Could not read {path}: {exc}")
return index
if __name__ == "__main__":
index = build_index("./evidence_copy")
with open("file_index.json", "w") as f:
json.dump(index, f, indent=2)
print(f"Indexed {len(index)} files")
Sample output, against a small evidence folder of six files
[2026-10-17T20:01:44.435] Indexed ./evidence_copy/History
[2026-10-17T20:01:44.440] Indexed ./evidence_copy/holiday.txt
[2026-10-17T20:01:44.443] Indexed ./evidence_copy/holiday_renamed.txt
[2026-10-17T20:01:44.445] Indexed ./evidence_copy/notes_invoice.pdf
[2026-10-17T20:01:44.446] Indexed ./evidence_copy/photo_001.jpg
[2026-10-17T20:01:44.448] Indexed ./evidence_copy/report.pdf
Indexed 6 files
Notice the log line for each file: that timestamped record is the first brick of the chain of custody, written automatically as the script works, using the logger from the last article. os.walk visits every folder under root_dir; for each file it records a path, a size, a true type and a hash, then appends that record to the growing index.
Python concept: The walrus operator :=
while chunk := f.read(chunk_size) both reads a chunk and tests it in one step. It reads a fixed-size piece of the file, assigns it to chunk, and the loop continues while that piece is non-empty. This hashes a multi-gigabyte file without ever loading the whole thing into memory, which an investigator's workstation will thank you for. You met the same shape of streaming read in Unit 1's file-handling article; this is the same idea put to forensic use.
Judging content, not the cover
Analogy: Judging content, not the cover
A file named holiday.jpg is not necessarily an image; a suspect can rename an executable to hide it. python-magic judges a file by its actual content, the signature bytes at its start, not by its extension, the way a reader judges a book by reading a page rather than trusting the title on the spine. This single habit catches one of the oldest and simplest evasions.
The evidence folder above includes holiday_renamed.txt, a real JPEG photograph saved with a .txt name, exactly the trick a suspect might try. Watch what the index records for it.
Searching the index
Once an index exists, searching it is a matter of filtering the list, which is far faster than walking the disk again for every question.
search_index.py
# search_index.py
import json
def search_by_type(index, mime_type):
return [r for r in index if r["mime_type"] == mime_type]
def search_by_keyword(index, keyword):
keyword = keyword.lower()
return [r for r in index if keyword in r["path"].lower()]
with open("file_index.json") as f:
index = json.load(f)
print(f"PDF files: {len(search_by_type(index, 'application/pdf'))}")
print(f"Files mentioning 'invoice': {len(search_by_keyword(index, 'invoice'))}")
# the renamed file: does python-magic see through the .txt extension?
images = search_by_type(index, 'image/jpeg')
print("Files identified as JPEG despite their name:")
for r in images:
print(" ", r["path"])
Sample output
PDF files: 2
Files mentioning 'invoice': 1
Files identified as JPEG despite their name:
./evidence_copy/holiday_renamed.txt
./evidence_copy/photo_001.jpg
holiday_renamed.txt shows up in the JPEG list despite its name, because detector.from_file() read the file's actual signature bytes, not its extension. A search by extension, or a search that trusted the file's name, would have missed it completely. This is the oldest and simplest evasion in digital forensics, and it is defeated by three lines of code.
Caution: Path separators differ by operating system
os.path.join and os.walk write paths using the operating system's own separator, a backslash on Windows and a forward slash on Linux and macOS. The index and search code above work identically either way; only the text of the path itself differs, which is worth remembering when you compare output between your own machine and someone else's.
Check your understanding
Question 1: Why does build_index.py identify a file's type with python-magic rather than by its extension? What evasion does this catch?
An extension is just text in the file's name, which anyone can change. python-magic reads the actual signature bytes at the start of the file's content, which do not change when the name does. This catches a suspect renaming a file, such as an image or an executable, to hide what it really is.
Question 2: What does hashing a file at collection time prove that simply recording its name and size does not?
A hash is a fingerprint of the file's exact content. Recording a name and size only shows that something with that name and size existed; two different files can easily share both. A hash proves the file's content has not changed since it was collected, which a name and size alone can never prove.
Question 3: Why does hash_file read the file in fixed-size chunks instead of reading it all at once?
Reading the whole file into memory at once would fail or become impractically slow for a very large file, such as a multi-gigabyte disk image. Reading fixed-size chunks and updating the hash incrementally processes the file a little at a time, so memory use stays small no matter how large the file is.
Key takeaway: Key takeaway
An index built once turns every later search into a filter over a list instead of a fresh walk of the disk. Hash every file as you index it, and judge its type by content, never by name: that one habit is what catches a suspect's oldest trick, the renamed file.