Selenium #

Selenium is a browser automation library that lets you control Chrome, Firefox, Edge, and other browsers programmatically — filling forms, clicking buttons, extracting data, and verifying the display exactly like a real user does. Its two main use cases are end-to-end testing (ensuring user flows work from browser to database) and web scraping (extracting data from pages that need JavaScript to render). This article uses Selenium 4 — its API changed significantly from version 3: the find_element_by_* methods were removed, replaced by the more consistent find_element(By.*).

Installation #

pip install selenium webdriver-manager

webdriver-manager handles the download and path of ChromeDriver/GeckoDriver automatically — no manual download or PATH configuration needed.

WebDriver Setup #

from selenium import webdriver
from selenium.webdriver.chrome.service  import Service as ChromeService
from selenium.webdriver.firefox.service import Service as FirefoxService
from selenium.webdriver.chrome.options  import Options as ChromeOptions
from webdriver_manager.chrome           import ChromeDriverManager
from webdriver_manager.firefox          import GeckoDriverManager

# ANTI-PATTERN: the old Selenium 3 way (executable_path deprecated)
driver = webdriver.Chrome(executable_path="/path/to/chromedriver")  # ✗

# CORRECT: Selenium 4 with webdriver-manager (auto-downloads the driver)
def buat_chrome_driver(headless: bool = False) -> webdriver.Chrome:
    options = ChromeOptions()

    if headless:
        options.add_argument("--headless=new")   # no-UI mode (for CI/CD)

    options.add_argument("--no-sandbox")
    options.add_argument("--disable-dev-shm-usage")
    options.add_argument("--window-size=1920,1080")
    options.add_argument("--disable-gpu")

    # Hide the sign that the browser is automated
    options.add_experimental_option("excludeSwitches", ["enable-automation"])
    options.add_experimental_option("useAutomationExtension", False)

    service = ChromeService(ChromeDriverManager().install())
    driver  = webdriver.Chrome(service=service, options=options)
    driver.implicitly_wait(0)   # turn off implicit wait -- use explicit waits only

    return driver

def buat_firefox_driver(headless: bool = False) -> webdriver.Firefox:
    options = webdriver.FirefoxOptions()
    if headless:
        options.add_argument("--headless")

    service = FirefoxService(GeckoDriverManager().install())
    return webdriver.Firefox(service=service, options=options)

# Usage
driver = buat_chrome_driver(headless=False)
try:
    driver.get("https://example.com")
    print(driver.title)
finally:
    driver.quit()   # always close the driver

Don’t use implicitly_wait together with explicit waits. They interact in unpredictable ways and can cause waits longer than expected. Pick one approach — explicit waits (WebDriverWait) are far better because they’re controlled per element.

Finding Elements #

Selenium 4 uses find_element(By.*) and find_elements(By.*). All find_element_by_* methods from Selenium 3 were removed.

from selenium.webdriver.common.by import By

driver.get("https://example.com/login")

# ANTI-PATTERN: old Selenium 3 methods (removed in Selenium 4)
driver.find_element_by_id("username")        # ✗ -- AttributeError
driver.find_element_by_xpath("//input")      # ✗ -- AttributeError

# CORRECT: Selenium 4
driver.find_element(By.ID,           "username")
driver.find_element(By.NAME,         "password")
driver.find_element(By.CLASS_NAME,   "btn-submit")
driver.find_element(By.TAG_NAME,     "h1")
driver.find_element(By.LINK_TEXT,    "Forgot Password?")
driver.find_element(By.PARTIAL_LINK_TEXT, "Forgot")
driver.find_element(By.XPATH,        "//button[@type='submit']")
driver.find_element(By.CSS_SELECTOR, "input[name='email']")
driver.find_element(By.CSS_SELECTOR, ".form-control.email-field")

# Fetch many elements at once
semua_link = driver.find_elements(By.TAG_NAME, "a")
produk_list = driver.find_elements(By.CSS_SELECTOR, ".produk-card")

for produk in produk_list:
    nama  = produk.find_element(By.CSS_SELECTOR, ".produk-nama").text
    harga = produk.find_element(By.CSS_SELECTOR, ".produk-harga").text
    print(f"{nama}: {harga}")

Locator Selection Strategy #

Locator preference order (from best to worst):

1. By.ID          -- fastest and most stable if a unique ID exists
2. By.CSS_SELECTOR -- fast, expressive, easier to read than XPath
3. By.XPATH       -- powerful for complex navigation, but slower
4. By.NAME        -- for form inputs with a name attribute
5. By.LINK_TEXT   -- specifically for <a> elements
6. By.CLASS_NAME  -- avoid if the class is generic (btn, form-control)
7. By.TAG_NAME    -- only if the tag is truly unique in that context

Avoid:
  ✗ Absolute XPath: /html/body/div[2]/div[1]/span  -- fragile, changes when HTML changes
  ✗ Generic classes: .container, .row, .col     -- can match many elements

Explicit Waits — Waiting for Elements #

Modern web pages use JavaScript that renders content asynchronously. You must wait for elements to be ready before interacting — don’t use wasteful time.sleep().

from selenium.webdriver.support.ui         import WebDriverWait
from selenium.webdriver.support            import expected_conditions as EC
from selenium.common.exceptions            import TimeoutException, NoSuchElementException

def tunggu_elemen(driver, by, locator, timeout: int = 10):
    """Wait for an element to appear and be clickable."""
    return WebDriverWait(driver, timeout).until(
        EC.element_to_be_clickable((by, locator))
    )

def tunggu_teks(driver, by, locator, teks: str, timeout: int = 10):
    """Wait for an element to contain a specific text."""
    return WebDriverWait(driver, timeout).until(
        EC.text_to_be_present_in_element((by, locator), teks)
    )

# Commonly used expected conditions
wait = WebDriverWait(driver, 10)

# Wait for an element to appear in the DOM (not necessarily visible)
wait.until(EC.presence_of_element_located((By.ID, "hasil")))

# Wait for an element to be visible and clickable
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[type='submit']")))

# Wait for an element to no longer be visible (loading spinner gone)
wait.until(EC.invisibility_of_element_located((By.CSS_SELECTOR, ".loading-spinner")))

# Wait for the URL to change
wait.until(EC.url_contains("/dashboard"))
wait.until(EC.url_matches(r"/dashboard/\d+"))

# Wait for the title to change
wait.until(EC.title_contains("Dashboard"))

# Handle timeout gracefully
try:
    elemen = WebDriverWait(driver, 5).until(
        EC.presence_of_element_located((By.ID, "popup-promo"))
    )
    elemen.find_element(By.CSS_SELECTOR, ".close-btn").click()
except TimeoutException:
    pass   # the popup didn't appear, continue

# ANTI-PATTERN: wasteful and unreliable time.sleep()
import time
time.sleep(3)                         # ✗ -- always waits 3 seconds even if the element is ready
elemen = driver.find_element(By.ID, "hasil")

# CORRECT: an efficient explicit wait
elemen = WebDriverWait(driver, 10).until(  # ✓ -- stops as soon as the element appears
    EC.presence_of_element_located((By.ID, "hasil"))
)

Interacting with Elements #

from selenium.webdriver.common.keys import Keys

# Fill and submit a form
email_field    = driver.find_element(By.ID, "email")
password_field = driver.find_element(By.ID, "password")

email_field.clear()
email_field.send_keys("[email protected]")

password_field.clear()
password_field.send_keys("password123")

# Submit with Enter
password_field.send_keys(Keys.RETURN)

# Or click the submit button
submit_btn = driver.find_element(By.CSS_SELECTOR, "button[type='submit']")
submit_btn.click()

# Read element content
judul   = driver.find_element(By.TAG_NAME, "h1").text
nilai   = driver.find_element(By.ID, "total").get_attribute("value")
href    = driver.find_element(By.CSS_SELECTOR, "a.btn-detail").get_attribute("href")
is_aktif = driver.find_element(By.ID, "checkbox-aktif").is_selected()

# Scroll to an element outside the viewport
elemen = driver.find_element(By.ID, "bagian-bawah")
driver.execute_script("arguments[0].scrollIntoView(true);", elemen)

# Click using JavaScript (useful if the element is covered by an overlay)
driver.execute_script("arguments[0].click();", elemen)

# Select from a dropdown (SELECT)
from selenium.webdriver.support.ui import Select

dropdown = Select(driver.find_element(By.ID, "kategori"))
dropdown.select_by_visible_text("Elektronik")
dropdown.select_by_value("elektronik")
dropdown.select_by_index(2)

ActionChains — Complex Interactions #

from selenium.webdriver.common.action_chains import ActionChains

actions = ActionChains(driver)

# Hover (mouse over)
menu_item = driver.find_element(By.CSS_SELECTOR, ".nav-item.dropdown")
actions.move_to_element(menu_item).perform()

# Click a submenu after hovering
submenu = WebDriverWait(driver, 5).until(
    EC.element_to_be_clickable((By.CSS_SELECTOR, ".dropdown-menu a:first-child"))
)
submenu.click()

# Drag and drop
sumber = driver.find_element(By.ID, "draggable")
tujuan = driver.find_element(By.ID, "droppable")
actions.drag_and_drop(sumber, tujuan).perform()

# Double click
elemen = driver.find_element(By.CSS_SELECTOR, ".item-editable")
actions.double_click(elemen).perform()

# Right click (context menu)
actions.context_click(elemen).perform()

# Keyboard shortcuts
from selenium.webdriver.common.keys import Keys
body = driver.find_element(By.TAG_NAME, "body")
actions.key_down(Keys.CONTROL).send_keys("a").key_up(Keys.CONTROL).perform()  # Ctrl+A

Page Object Model #

The Page Object Model (POM) is an architecture pattern separating page interaction logic from test logic. Each web page is represented as a Python class — UI changes only need to be updated in one place.

# pages/base_page.py

from selenium.webdriver.support.ui  import WebDriverWait
from selenium.webdriver.support     import expected_conditions as EC
from selenium.common.exceptions     import TimeoutException

class BasePage:
    def __init__(self, driver):
        self.driver  = driver
        self.wait    = WebDriverWait(driver, 10)
        self.timeout = 10

    def buka(self, url: str):
        self.driver.get(url)

    def temukan(self, by, locator):
        return self.wait.until(EC.presence_of_element_located((by, locator)))

    def klik(self, by, locator):
        self.wait.until(EC.element_to_be_clickable((by, locator))).click()

    def isi(self, by, locator, teks: str):
        elemen = self.temukan(by, locator)
        elemen.clear()
        elemen.send_keys(teks)

    def ambil_teks(self, by, locator) -> str:
        return self.temukan(by, locator).text

    def screenshot(self, nama_file: str):
        self.driver.save_screenshot(f"screenshots/{nama_file}.png")

    def tunggu_url_berubah(self, url_fragment: str):
        self.wait.until(EC.url_contains(url_fragment))

# pages/login_page.py

from selenium.webdriver.common.by import By
from .base_page import BasePage

class LoginPage(BasePage):
    # Locators defined in the class, not in tests
    URL            = "https://myapp.com/login"
    EMAIL_INPUT    = (By.ID,          "email")
    PASSWORD_INPUT = (By.ID,          "password")
    SUBMIT_BTN     = (By.CSS_SELECTOR, "button[type='submit']")
    ERROR_MSG      = (By.CSS_SELECTOR, ".alert-danger")
    REMEMBER_ME    = (By.ID,          "remember-me")

    def buka_halaman_login(self):
        self.buka(self.URL)

    def login(self, email: str, password: str, ingat: bool = False):
        self.isi(*self.EMAIL_INPUT, email)
        self.isi(*self.PASSWORD_INPUT, password)
        if ingat:
            self.klik(*self.REMEMBER_ME)
        self.klik(*self.SUBMIT_BTN)

    def ambil_pesan_error(self) -> str:
        try:
            return self.ambil_teks(*self.ERROR_MSG)
        except Exception:
            return ""

    def adalah_halaman_login(self) -> bool:
        return "/login" in self.driver.current_url

# pages/dashboard_page.py

from selenium.webdriver.common.by import By
from .base_page import BasePage

class DashboardPage(BasePage):
    WELCOME_MSG    = (By.CSS_SELECTOR, ".welcome-message")
    LOGOUT_BTN     = (By.ID,           "btn-logout")
    PRODUK_COUNT   = (By.CSS_SELECTOR, ".stat-produk .count")

    def tunggu_dashboard_muncul(self):
        self.tunggu_url_berubah("/dashboard")

    def ambil_pesan_selamat_datang(self) -> str:
        return self.ambil_teks(*self.WELCOME_MSG)

    def logout(self):
        self.klik(*self.LOGOUT_BTN)

    def ambil_jumlah_produk(self) -> int:
        return int(self.ambil_teks(*self.PRODUK_COUNT))

Integration with PyTest #

# tests/conftest.py

import pytest
import os
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome          import ChromeDriverManager
from selenium.webdriver.chrome.options import Options

@pytest.fixture(scope="module")
def driver():
    """Chrome driver -- created once per test module."""
    options = Options()
    options.add_argument("--headless=new")
    options.add_argument("--no-sandbox")
    options.add_argument("--disable-dev-shm-usage")
    options.add_argument("--window-size=1920,1080")

    service = Service(ChromeDriverManager().install())
    driver  = webdriver.Chrome(service=service, options=options)
    driver.implicitly_wait(0)

    yield driver

    driver.quit()

@pytest.fixture(autouse=True)
def screenshot_on_failure(driver, request):
    """Automatic screenshot when a test fails."""
    yield
    if request.node.rep_call.failed if hasattr(request.node, "rep_call") else False:
        os.makedirs("screenshots", exist_ok=True)
        nama = request.node.name.replace(" ", "_")
        driver.save_screenshot(f"screenshots/FAIL_{nama}.png")

@pytest.hookimpl(tryfirst=True, hookwrapper=True)
def pytest_runtest_makereport(item, call):
    """Hook to detect test failures."""
    outcome = yield
    rep     = outcome.get_result()
    setattr(item, "rep_" + rep.when, rep)

# tests/test_login.py

import pytest
from pages.login_page     import LoginPage
from pages.dashboard_page import DashboardPage

BASE_URL = "https://myapp.com"

@pytest.fixture
def login_page(driver):
    page = LoginPage(driver)
    page.buka_halaman_login()
    return page

@pytest.fixture
def dashboard_page(driver):
    return DashboardPage(driver)

def test_login_berhasil(login_page, dashboard_page):
    login_page.login("[email protected]", "password123")
    dashboard_page.tunggu_dashboard_muncul()

    assert "/dashboard" in login_page.driver.current_url
    pesan = dashboard_page.ambil_pesan_selamat_datang()
    assert "Selamat datang" in pesan

def test_login_password_salah(login_page):
    login_page.login("[email protected]", "passwordsalah")

    pesan_error = login_page.ambil_pesan_error()
    assert "Email atau password salah" in pesan_error
    assert login_page.adalah_halaman_login()

def test_login_email_kosong(login_page):
    login_page.login("", "password123")

    pesan_error = login_page.ambil_pesan_error()
    assert pesan_error or login_page.adalah_halaman_login()

@pytest.mark.parametrize("email,password", [
    ("",               "password"),
    ("bukan-email",    "password"),
    ("[email protected]",  ""),
])
def test_login_input_tidak_valid(login_page, email, password):
    login_page.login(email, password)
    assert login_page.adalah_halaman_login()

Web Scraping with Selenium #

Selenium is used for scraping when content is rendered by JavaScript and can’t be fetched with plain requests.

from selenium import webdriver
from selenium.webdriver.common.by      import By
from selenium.webdriver.support.ui     import WebDriverWait
from selenium.webdriver.support        import expected_conditions as EC
import json, time

def scrape_produk_ecommerce(url: str) -> list[dict]:
    driver = buat_chrome_driver(headless=True)
    produk_list = []

    try:
        driver.get(url)

        # Wait for the first product to appear
        WebDriverWait(driver, 15).until(
            EC.presence_of_element_located((By.CSS_SELECTOR, ".product-card"))
        )

        # Scroll down to load lazy-loaded content
        tinggi_lama = 0
        while True:
            driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
            time.sleep(1.5)
            tinggi_baru = driver.execute_script("return document.body.scrollHeight")
            if tinggi_baru == tinggi_lama:
                break
            tinggi_lama = tinggi_baru

        # Extract data from all products
        cards = driver.find_elements(By.CSS_SELECTOR, ".product-card")
        for card in cards:
            try:
                produk_list.append({
                    "nama":  card.find_element(By.CSS_SELECTOR, ".product-name").text.strip(),
                    "harga": card.find_element(By.CSS_SELECTOR, ".product-price").text.strip(),
                    "rating": card.find_element(By.CSS_SELECTOR, ".product-rating").get_attribute("data-score"),
                    "url":   card.find_element(By.CSS_SELECTOR, "a").get_attribute("href"),
                })
            except Exception:
                continue   # skip incomplete products

    finally:
        driver.quit()

    return produk_list

# Save the results to JSON
hasil = scrape_produk_ecommerce("https://contoh-toko.com/produk")
with open("produk.json", "w", encoding="utf-8") as f:
    json.dump(hasil, f, ensure_ascii=False, indent=2)
print(f"{len(hasil)} products scraped successfully.")

Summary #

The new Selenium 4 API — use find_element(By.ID, "...") not find_element_by_id(); all find_element_by_* methods were removed in Selenium 4.
webdriver-manager — use it for auto-download and management of ChromeDriver/GeckoDriver; no manual download or PATH configuration needed.
Explicit waits, not time.sleep() — use WebDriverWait with the right expected_conditions; faster and more reliable because it stops as soon as the condition is met.
Turn off implicitly_wait — set it to 0 and use only explicit waits; mixing them causes unpredictable behavior.
CSS Selectors over XPath — easier to read and generally faster; use XPath only for complex DOM navigation or when CSS isn’t enough.
Page Object Model for test suites — separate locators and interactions into dedicated page classes; UI changes only need updating in one place.
driver.quit() is mandatory in finally — always close the driver in a finally block or via a pytest fixture so browsers don’t leak even when tests fail.
--headless=new — use the latest headless flag for Chrome in CI/CD; more stable than the old --headless.
Screenshots on test failure — implement the pytest pytest_runtest_makereport hook for automatic screenshots when tests fail; very helpful for CI debugging.
Scroll for lazy-loaded content — use execute_script("window.scrollTo...") to load content that appears on scroll, especially when scraping.

← Previous: PyTest Next: Articles & Resources →