Regex #

A regular expression (regex) is a mini-language for describing patterns in text. With regex you can find, extract, validate, and replace text based on highly flexible patterns — far more powerful than plain string methods like split() or startswith(). Python provides the re module in the stdlib for all regex operations. What makes regex powerful and tricky at the same time is that the same pattern can mean many things depending on context — greedy vs non-greedy, anchored vs floating, capturing vs non-capturing. This article covers all the important regex concepts in Python with real examples.

Raw Strings for Regex Patterns #

Before writing patterns, it’s important to understand why regex always uses raw strings r"...":

import re

# ANTI-PATTERN: a regular string — Python interprets the backslash first
pattern = "\d+"          # Python interprets \d as d (unknown escape)
                        # the actual pattern is just "d+"

# CORRECT: a raw string — Python doesn't interpret the backslash
pattern = r"\d+"         # \d is passed to the regex engine as-is
pattern = r"\b\w+\b"     # word boundary + word chars + word boundary

# Without a raw string you must double-escape every backslash
pattern_no_raw = "\\d+"   # same as r"\d+" but harder to read

Characters and Character Classes #

# Literal characters — match exactly
re.findall(r"cat", "the cat sat on the mat")   # → ['cat']

# . (dot) — matches ANY character except newline
re.findall(r"c.t", "cat cut c4t c\nt")   # → ['cat', 'cut', 'c4t']

# Character classes [...] — match ONE character from the set
re.findall(r"[aeiou]", "Python")          # → ['o']
re.findall(r"[a-z]", "Hello World")       # → ['e', 'l', 'l', 'o', 'o', 'r', 'l', 'd']
re.findall(r"[A-Z]", "Hello World")       # → ['H', 'W']
re.findall(r"[0-9]", "abc123def")         # → ['1', '2', '3']
re.findall(r"[a-zA-Z0-9]", "Hi! 42")     # → ['H', 'i', '4', '2']

# [^...] — match characters NOT in the set
re.findall(r"[^aeiou]", "Python")         # → ['P', 'y', 't', 'h', 'n']
re.findall(r"[^0-9]", "abc123")           # → ['a', 'b', 'c']

Shorthand Character Classes #

Shorthand	Equivalent to	Meaning
`\d`	`[0-9]`	digit
`\D`	`[^0-9]`	not a digit
`\w`	`[a-zA-Z0-9_]`	word character (alphanumeric + _)
`\W`	`[^a-zA-Z0-9_]`	not a word character
`\s`	`[ \t\n\r\f\v]`	whitespace (space, tab, newline)
`\S`	`[^ \t\n\r\f\v]`	not whitespace

text = "Budi123 bought 5 apples"

print(re.findall(r"\d+", text))    # → ['123', '5']
print(re.findall(r"\w+", text))    # → ['Budi123', 'bought', '5', 'apples']
print(re.findall(r"\s+", text))    # → [' ', ' ', ' ']

Quantifiers — Controlling How Many Times #

text = "aababaabbb"

# * — 0 or more
re.findall(r"ab*", text)    # → ['a', 'ab', 'a', 'ab', 'a', 'ab', 'abbb'] — hmm

# + — 1 or more
re.findall(r"\d+", "a1b22c333")    # → ['1', '22', '333']

# ? — 0 or 1 (optional)
re.findall(r"colou?r", "color colour")   # → ['color', 'colour']

# {n} — exactly n times
re.findall(r"\d{3}", "12 123 1234")    # → ['123', '123']  (from 1234)

# {n,m} — between n and m times
re.findall(r"\d{2,4}", "1 12 123 1234 12345")   # → ['12', '123', '1234', '1234']

# {n,} — n or more
re.findall(r"\d{3,}", "12 123 1234 12345")   # → ['123', '1234', '12345']

Greedy vs Non-Greedy #

By default, quantifiers are greedy — matching as many characters as possible. Add ? after a quantifier to make it non-greedy (matching as few as possible):

html = "<b>bold</b> and <i>italic</i>"

# Greedy — matches as much as possible
print(re.findall(r"<.+>", html))
# → ['<b>bold</b> and <i>italic</i>']  ← one big match

# Non-greedy — matches as little as possible
print(re.findall(r"<.+?>", html))
# → ['<b>', '</b>', '<i>', '</i>']  ← each tag separately

# Another example
text = '"apple", "orange", "mango"'

# Greedy — from the first quote to the LAST quote
re.findall(r'".*"', text)    # → ['"apple", "orange", "mango"']

# Non-greedy — each quoted word
re.findall(r'".*?"', text)   # → ['"apple"', '"orange"', '"mango"']

Anchors — Positions in the String #

# ^ — start of the string (or start of each line with re.MULTILINE)
re.match(r"^Hello", "Hello World")    # → match
re.match(r"^World", "Hello World")    # → None

# $ — end of the string (or end of each line with re.MULTILINE)
re.search(r"World$", "Hello World")   # → match
re.search(r"Hello$", "Hello World")   # → None

# \b — word boundary (the edge between \w and \W)
print(re.findall(r"\bcat\b", "the cat scattered concatenate"))
# → ['cat']  ← only the standalone word "cat", not "cat" inside other words

print(re.findall(r"cat", "the cat scattered concatenate"))
# → ['cat', 'cat', 'cat']  ← matches inside words too

# \A — start of the string (stricter than ^, unaffected by MULTILINE)
# \Z — end of the string (stricter than $)
re.match(r"\APython", "Python is great")   # → match

The `re` Functions #

The re module provides various functions for searching, matching, and manipulating strings. To understand how text processing with regular expressions works end-to-end — from the raw pattern to result extraction — look at the flow diagram below:

flowchart LR
    PolaMentah["Raw Pattern (Raw String r'...')"] --> Compile["re.compile() (Recommended)"]
    Compile --> PatternObj["Compiled Pattern Object"]
    
    PatternObj --> SearchCall["Choose a Search Method:<br/>- match()<br/>- search()<br/>- finditer()<br/>- findall()"]
    
    SearchCall --> MatchEngine["Run the Search Engine"]
    MatchEngine --> MatchObj["Match Object (re.Match)"]
    
    MatchObj --> Extract["Extract the Results:<br/>- group() / group(i)<br/>- groups()<br/>- groupdict()"]

With this flow, the search pattern is translated by the regex engine into a series of fast matching instructions, then the results are wrapped in a Match Object so you can access the information.

`re.match()` vs `re.search()` #

import re

text = "Budi Santoso is 28 years old"

# match() — only matches at the START of the string
print(re.match(r"\d+", text))         # → None  (the string doesn't start with a digit)
print(re.match(r"Budi", text))        # → match

# search() — searches ANYWHERE in the string
print(re.search(r"\d+", text))        # → match  (28 found)
print(re.search(r"\d+", text).group())  # → '28'

# ANTI-PATTERN: using match() when you should use search()
# match() is only suitable for validating a pattern at the start of the string

`re.findall()` and `re.finditer()` #

text = "Emails: [email protected], [email protected], [email protected]"

# findall() — return all matches as a list of strings
email_list = re.findall(r"\b[\w.-]+@[\w.-]+\.\w+\b", text)
print(email_list)
# → ['[email protected]', '[email protected]', '[email protected]']

# finditer() — return an iterator of Match objects (more efficient for large data)
for m in re.finditer(r"\b[\w.-]+@[\w.-]+\.\w+\b", text):
    print(f"  Email: {m.group()}, position: {m.start()}–{m.end()}")

# findall with groups () — returns a list of tuples if there are groups
text = "Names: Budi (28), Ani (32), Citra (25)"
result = re.findall(r"(\w+) \((\d+)\)", text)
print(result)   # → [('Budi', '28'), ('Ani', '32'), ('Citra', '25')]

`re.sub()` — Replacing Patterns #

text = "Price: Rp 25.000 and Rp 150.000"

# Replace numbers with a placeholder
print(re.sub(r"\d+", "X", text))
# → Price: Rp X.X and Rp X.X

# Limit replacements with count
print(re.sub(r"\d+", "X", text, count=2))
# → Price: Rp X.X and Rp 150.000

# Use groups in the replacement — \1, \2 to reference groups
name = "Santoso, Budi"
swapped = re.sub(r"(\w+), (\w+)", r"\2 \1", name)
print(swapped)   # → Budi Santoso

# Use a function as the replacement
def to_upper(m):
    return m.group().upper()

print(re.sub(r"\b\w{5,}\b", to_upper, "Python is a great language"))
# → PYTHON is a GREAT LANGUAGE  (only words ≥5 letters)

`re.split()` — Splitting Strings #

# Split on whitespace (one or more)
print(re.split(r"\s+", "  one   two    three  "))
# → ['', 'one', 'two', 'three', '']

# Split on several delimiters at once
text = "apple,orange;mango|banana"
print(re.split(r"[,;|]", text))
# → ['apple', 'orange', 'mango', 'banana']

# Keep the delimiters in the result (use a group)
print(re.split(r"([,;|])", text))
# → ['apple', ',', 'orange', ';', 'mango', '|', 'banana']

Capturing Groups #

Groups let you extract specific parts of a match:

# Basic groups with ()
pattern = r"(\d{4})-(\d{2})-(\d{2})"
text = "Date: 2024-08-17"

m = re.search(pattern, text)
if m:
    print(m.group())     # → 2024-08-17  (the whole match)
    print(m.group(1))    # → 2024        (group 1)
    print(m.group(2))    # → 08          (group 2)
    print(m.group(3))    # → 17          (group 3)
    print(m.groups())    # → ('2024', '08', '17')  (all groups)

Named Groups #

Named groups make patterns easier to read and use:

# (?P<name>...) — a group with a name
pattern = r"(?P<year>\d{4})-(?P<month>\d{2})-(?P<day>\d{2})"
text = "2024-08-17"

m = re.match(pattern, text)
if m:
    print(m.group("year"))    # → 2024
    print(m.group("month"))   # → 08
    print(m.group("day"))     # → 17
    print(m.groupdict())      # → {'year': '2024', 'month': '08', 'day': '17'}

# Named groups in sub() — reference with \g<name>
log = "2024-08-17 09:30:45 - INFO - Server started"
pattern = r"(?P<date>\d{4}-\d{2}-\d{2}) (?P<time>\d{2}:\d{2}:\d{2})"
result = re.sub(pattern, r"[\g<date> \g<time>]", log)
print(result)   # → [2024-08-17 09:30:45] - INFO - Server started

Non-Capturing Groups #

# (?:...) — a group for grouping but NOT captured
pattern = r"(?:https?|ftp)://(\w+\.\w+)"
text = "Visit https://python.org or http://docs.python.org"

# findall with a non-capturing group only returns the captured groups
result = re.findall(pattern, text)
print(result)   # → ['python.org', 'docs.python.org']
# The protocol (https/http) isn't captured because of (?:...)

Lookahead and Lookbehind #

Lookahead and lookbehind match a position based on what surrounds it — without including that text in the match:

# Positive lookahead (?=...) — match if FOLLOWED by the pattern
prices = "Rp 25000 USD 50 EUR 75"
# Numbers followed by " USD"
print(re.findall(r"\d+(?= USD)", prices))   # → ['50']

# Negative lookahead (?!...) — match if NOT followed by the pattern
print(re.findall(r"\d+(?! USD)", prices))   # → ['25000', '75']  (not USD numbers)

# Positive lookbehind (?<=...) — match if PRECEDED by the pattern
# Numbers preceded by "Rp "
print(re.findall(r"(?<=Rp )\d+", prices))  # → ['25000']

# Negative lookbehind (?<!...) — match if NOT preceded by the pattern
print(re.findall(r"(?<!Rp )\d+", prices))  # → ['50', '75']

# Practical example: extracting values from password requirements
# Find words containing BOTH an uppercase letter and a digit
word_list = ["Password1", "password", "PASSWORD", "Pass123", "123456"]

def has_upper_and_digit(word):
    return bool(re.search(r"(?=.*[A-Z])(?=.*\d)", word))

for word in word_list:
    print(f"{word}: {has_upper_and_digit(word)}")
# → Password1: True
# → password: False
# → PASSWORD: False
# → Pass123: True
# → 123456: False

`re.compile()` — Compiling Patterns for Performance #

If you use the same pattern repeatedly, compile it first with re.compile():

import re

# ANTI-PATTERN: recompiling every iteration — slow for large data
text_list = [...]
for text in text_list:
    result = re.findall(r"\b[\w.-]+@[\w.-]+\.\w+\b", text)

# CORRECT: compile once, use many times
EMAIL_PATTERN = re.compile(r"\b[\w.-]+@[\w.-]+\.\w+\b")

for text in text_list:
    result = EMAIL_PATTERN.findall(text)   # uses the compiled pattern

# All the re functions are available as methods on the compiled object
m = EMAIL_PATTERN.search("Send to [email protected] please")
if m:
    print(m.group())

# Another benefit of compiling: you can add comments with re.VERBOSE
PHONE_PATTERN = re.compile(r"""
    (?:\+62|0)      # country code or 0
    [\s-]?          # optional space or dash
    (?:8\d{2}|21)   # area code: 8xx (mobile) or 21 (Jakarta)
    [\s-]?          # optional separator
    \d{3,4}         # number part 1
    [\s-]?          # optional separator
    \d{3,4}         # number part 2
""", re.VERBOSE)

Flags #

# re.IGNORECASE (re.I) — case insensitive
print(re.findall(r"python", "Python PYTHON python", re.I))
# → ['Python', 'PYTHON', 'python']

# re.MULTILINE (re.M) — ^ and $ match the start/end of each line
multi_text = "first line\nsecond line\nthird line"
print(re.findall(r"^line", multi_text, re.M))
# → ['line', 'line', 'line']  (without M only 1)

# re.DOTALL (re.S) — . also matches newlines
text = "start\nmiddle\nend"
print(re.search(r"start.+end", text))           # → None (. doesn't match \n)
print(re.search(r"start.+end", text, re.S))     # → match

# Combining flags
print(re.findall(r"^python", multi_text, re.I | re.M))

# re.VERBOSE (re.X) — allow comments and whitespace in patterns
EMAIL_PATTERN = re.compile(r"""
    [\w.+-]+        # local part — letters, digits, dots, plus, dash
    @               # literal @ character
    [\w-]+          # domain name
    (?:\.[\w-]+)*   # sub-domains (optional, can be more than one)
    \.\w{2,}        # TLD — at least 2 characters
""", re.VERBOSE)

Common Validation Patterns #

import re

# Simple email
EMAIL_PATTERN = re.compile(r"^[\w.+-]+@[\w-]+\.\w{2,}$")

# Indonesian phone number
PHONE_ID_PATTERN = re.compile(r"^(?:\+62|0)[2-9]\d{7,11}$")

# URL
URL_PATTERN = re.compile(
    r"https?://"          # protocol
    r"(?:[\w-]+\.)*"      # optional subdomains
    r"[\w-]+\.\w{2,}"     # domain + TLD
    r"(?:/[^\s]*)?"       # optional path
)

# Indonesian national ID (16 digits)
NIK_PATTERN = re.compile(r"^\d{16}$")

# Indonesian postal code (5 digits)
POSTAL_CODE_PATTERN = re.compile(r"^\d{5}$")

# Date in YYYY-MM-DD format (format validation only, not calendar)
DATE_PATTERN = re.compile(
    r"^\d{4}"            # year
    r"-(0[1-9]|1[0-2])"  # month 01–12
    r"-(0[1-9]|[12]\d|3[01])$"  # day 01–31
)

# Usage
def validate_email(email: str) -> bool:
    return bool(EMAIL_PATTERN.match(email))

def validate_phone(phone: str) -> bool:
    # Normalize first — remove spaces and dashes
    clean_phone = re.sub(r"[\s-]", "", phone)
    return bool(PHONE_ID_PATTERN.match(clean_phone))

print(validate_email("[email protected]"))   # → True
print(validate_email("not-an-email"))       # → False
print(validate_phone("+62 812-3456-7890"))  # → True
print(validate_phone("1234"))               # → False

When NOT to Use Regex #

Regex is very powerful, but it isn’t always the best solution:

# Use str methods when the pattern is simple and literal
text = "Hello, World!"

# ANTI-PATTERN: regex for simple operations
if re.match(r"^Hello", text):  # overkill
    pass
if re.search(r"World", text):  # overkill
    pass

# CORRECT: str methods are faster and easier to read
if text.startswith("Hello"):
    pass
if "World" in text:
    pass

# Don't parse HTML/XML with regex — use the right parser
# ANTI-PATTERN:
title = re.findall(r"<title>(.*?)</title>", html)  # fragile and limited

# CORRECT: use BeautifulSoup or lxml
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, "html.parser")
title = soup.find("title").text

Use regex when:
  ✓ The pattern can't be expressed with plain string operations
  ✓ You need to extract several parts from text at once
  ✓ The pattern involves variation (optional, repetition, alternation)
  ✓ Validating complex formats (email, URL, national ID)

Use str methods when:
  ✓ Checking a prefix/suffix: startswith(), endswith()
  ✓ Checking existence: 'x' in text
  ✓ Replacing a fixed substring: text.replace("old", "new")
  ✓ Splitting on a fixed delimiter: text.split(",")
  ✓ Removing whitespace: text.strip()

Summary #

Always use raw strings r"..." for regex patterns — this stops Python from interpreting backslashes before they reach the regex engine.
re.search() not re.match() for searching inside a string — match() only matches at the start of the string.
Greedy vs non-greedy — add ? after a quantifier (+?, *?) to match as few characters as possible.
Named groups (?P<name>...) make patterns easier to read and let you access results by name instead of index.
Non-capturing groups (?:...) for grouping without capturing — they don’t affect findall() and groups() results.
re.compile() for patterns used repeatedly — avoids recompiling on every call.
re.VERBOSE for long patterns — allows whitespace and comments inside the pattern for readability.
Lookahead/lookbehind to match based on surrounding context without including that context in the match.
Don’t use regex for simple string operations — startswith(), in, split(), and replace() are faster and easier to read.
Don’t parse HTML/XML with regex — use BeautifulSoup or lxml, which understand document structure.

← Previous: Date & Time Next: Virtual Environments →