Unit 2 classified phishing URLs using hand-engineered features (URL length, digit ratio, has-IP-address). There is a second, complementary approach worth knowing: treat the URL itself as text, tokenise it, and let TF-IDF discover the discriminative substrings automatically, exactly the way Unit 1 and this unit treat email and message text.
url_tokenize.py
def url_tokenize(url: str) -> list[str]:
"""Split a URL into its slash-, dot-, and hyphen-separated parts,
the same token boundaries a human skimming a URL actually uses."""
url = url.lower()
tokens = []
for segment in url.split("/"):
for part in segment.split("-"):
tokens.extend(part.split("."))
tokens = [t for t in set(tokens) if t and t != "com"]
return tokens
url_tfidf_classifier.py
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
url_vectorizer = TfidfVectorizer(tokenizer=url_tokenize)
X_urls = url_vectorizer.fit_transform(url_list)
url_model = LogisticRegression(max_iter=2000)
url_model.fit(X_urls, labels)
# scoring a brand-new URL at prediction time
new_urls = ["paypal-secure-login.account-verify.ru", "github.com"]
predictions = url_model.predict(url_vectorizer.transform(new_urls))
ML concept: Lexical versus Structured URL Features: Which to Use?
Hand-engineered structured features (Unit 2) are interpretable and cheap to compute, but require a human to think of every useful pattern in advance. Lexical TF-IDF features discover patterns automatically, including ones a human would never think to hand-code, such as a specific brand name appearing in a subdomain it has no business being in, but are harder to explain to a non-technical stakeholder and can overfit to coincidental vocabulary in a particular training set. Production phishing-URL classifiers commonly use both feature families together, exactly as this unit's phishing-email pipeline later combines text and structural features.
Try it yourself: Tokenise It Yourself
Trace url_tokenize("paypal-secure-login.account-verify.ru") by hand, segment by segment, and list the final set of tokens it would produce. Which of those tokens would you expect a trained classifier to weight most heavily toward phishing, and why?