IO #

Almost every real program needs to interact with the file system — reading configuration, saving results, processing CSV data, or writing logs. Python provides several layers for these operations: the built-in open() function for direct file access, the pathlib module for modern path navigation, the os and shutil modules for file system manipulation, and the csv and json modules for common data formats. Understanding when to use each will make your code cleaner and safer.

To make it easier to decide which module to use for each I/O operation, consider the following decision flow:

flowchart TD
    Task["IO Operation Need"] --> PathOps{"Path Manipulation / File Check?"}
    PathOps -->|"Yes"| Pathlib["Use pathlib.Path (Modern & Cross-Platform)"]
    PathOps -->|"No"| FileOps{"Reading / Writing File Contents?"}
    FileOps -->|"Yes"| OpenWith["Use open() with a with statement"]
    FileOps -->|"No"| DirOps{"Copy, Move, or Delete Files/Dirs?"}
    DirOps -->|"Yes"| OsShutil["Use shutil or os (File System Operations)"]
    DirOps -->|"No"| DataOps{"CSV / JSON Format?"}
    DataOps -->|"Yes"| CsvJson["Use the csv or json module (Data Parsing)"]

Reading and Writing Files #

The open() function is the main gateway to working with files. Always use a with statement so the file is automatically closed even if an error occurs.

# ANTI-PATTERN: opening a file without with
f = open("data.txt", "r")
konten = f.read()
f.close()   # -- if an error occurs before this, the file is never closed

# CORRECT: use a with statement
with open("data.txt", "r") as f:
    konten = f.read()
# the file is automatically closed here, even if an exception occurs

File Opening Modes #

# The most frequently used modes:
open("file.txt", "r")    # read -- default, the file must exist
open("file.txt", "w")    # write -- create new or overwrite contents
open("file.txt", "a")    # append -- add at the end, create if missing
open("file.txt", "x")    # exclusive -- create new, error if it exists
open("file.txt", "rb")   # read binary
open("file.txt", "wb")   # write binary

Reading Files #

# Read the whole contents at once
with open("data.txt", "r", encoding="utf-8") as f:
    isi = f.read()

# Read line by line -- efficient for large files
with open("data.txt", "r", encoding="utf-8") as f:
    for baris in f:
        print(baris.rstrip("\n"))

# Read all lines into a list
with open("data.txt", "r", encoding="utf-8") as f:
    semua_baris = f.readlines()

# readline() -- read one line
with open("data.txt", "r", encoding="utf-8") as f:
    baris_pertama = f.readline()

Writing Files #

baris = ["baris pertama", "baris kedua", "baris ketiga"]

# Write a string
with open("output.txt", "w", encoding="utf-8") as f:
    f.write("Hello, world!\n")

# Write many lines at once
with open("output.txt", "w", encoding="utf-8") as f:
    f.writelines(f"{b}\n" for b in baris)

# Append -- add to the end of an existing file
with open("log.txt", "a", encoding="utf-8") as f:
    f.write("New log entry\n")
Always specify encoding="utf-8" explicitly when opening text files. The default encoding varies by operating system — Windows often uses cp1252 or latin-1, which can cause errors when the text contains non-ASCII characters like accented letters.

Reading Binary Files #

# Read an image file or other binary file
with open("gambar.png", "rb") as f:
    data = f.read()

# Manually copy a binary file
with open("sumber.bin", "rb") as src, open("tujuan.bin", "wb") as dst:
    while chunk := src.read(8192):   # read per 8KB
        dst.write(chunk)

pathlib — Modern Path Navigation #

The pathlib module (Python 3.4+) is the recommended way to work with file system paths. It models paths as objects, not strings, making path operations more intuitive and cross-platform.

from pathlib import Path

# Create a Path object
p = Path("dokumen/laporan.txt")
home = Path.home()          # the user's home directory
cwd = Path.cwd()            # the current working directory

# Path navigation with the / operator
project_dir = Path("/home/user/project")
config_file = project_dir / "config" / "settings.json"
print(config_file)  # /home/user/project/config/settings.json

Useful Path Properties #

p = Path("/home/user/dokumen/laporan.pdf")

print(p.name)       # "laporan.pdf"       -- full file name
print(p.stem)       # "laporan"           -- name without the extension
print(p.suffix)     # ".pdf"              -- the extension
print(p.suffixes)   # [".pdf"]            -- all extensions (for .tar.gz → ['.tar', '.gz'])
print(p.parent)     # /home/user/dokumen  -- the parent directory
print(p.parts)      # ('/', 'home', 'user', 'dokumen', 'laporan.pdf')

File Operations with pathlib #

from pathlib import Path

p = Path("data.txt")

# Check existence
print(p.exists())       # True / False
print(p.is_file())      # True if a file
print(p.is_dir())       # True if a directory

# Read and write directly
p.write_text("File contents", encoding="utf-8")
isi = p.read_text(encoding="utf-8")

p_biner = Path("data.bin")
p_biner.write_bytes(b"\x00\x01\x02")
data = p_biner.read_bytes()

# Rename / move
p.rename(Path("data_baru.txt"))

# Delete a file
p.unlink()
p.unlink(missing_ok=True)   # no error if the file doesn't exist (Python 3.8+)

Creating and Deleting Directories #

from pathlib import Path

# Create a directory
Path("direktori_baru").mkdir()
Path("a/b/c").mkdir(parents=True, exist_ok=True)  # create all levels, no error if it exists

# Delete an empty directory
Path("direktori_baru").rmdir()

# List directory contents
p = Path(".")
for item in p.iterdir():
    print(item)

# Glob -- find files with a pattern
for f in Path(".").glob("*.txt"):
    print(f)

for f in Path(".").rglob("*.py"):   # recursive
    print(f)
Use pathlib for all path operations in new code. It’s cleaner than manual string concatenation (os.path.join()), automatically cross-platform, and works directly with open().

The os Module — OS Interaction #

The os module provides access to low-level operating system functions. For path manipulation, pathlib is more recommended, but os is still needed for some operations like environment variables and process information.

import os

# Environment variables
home_dir = os.environ.get("HOME", "/tmp")
db_url = os.environ.get("DATABASE_URL")   # None if not present

# Working directory
print(os.getcwd())       # current directory
os.chdir("/tmp")         # change directory

# File information
stat = os.stat("data.txt")
print(stat.st_size)      # file size in bytes
print(stat.st_mtime)     # last modification time (Unix timestamp)

# Run a system command
exit_code = os.system("ls -la")

os.path for Path Operations #

import os

# ANTI-PATTERN: joining paths with strings
path = "/home/user" + "/" + "dokumen" + "/" + "file.txt"

# CORRECT: use os.path.join() or better, pathlib
path = os.path.join("/home/user", "dokumen", "file.txt")

print(os.path.exists(path))          # check existence
print(os.path.isfile(path))          # check whether it's a file
print(os.path.isdir("/home/user"))   # check whether it's a directory
print(os.path.basename(path))        # "file.txt"
print(os.path.dirname(path))         # "/home/user/dokumen"
print(os.path.splitext("file.txt"))  # ("file", ".txt")
print(os.path.getsize(path))         # size in bytes

File and Directory Manipulation with os #

import os

os.mkdir("direktori_baru")                    # create a directory (one level)
os.makedirs("a/b/c", exist_ok=True)          # create all levels
os.rename("lama.txt", "baru.txt")            # rename / move
os.remove("file_yang_mau_dihapus.txt")       # delete a file
os.rmdir("direktori_kosong")                 # delete an empty directory
isi = os.listdir(".")                        # list directory contents

The shutil Module — High-Level File Operations #

shutil (shell utilities) provides functions for more complex operations like copying directories with their contents or deleting non-empty directories.

import shutil

# Copy files
shutil.copy("sumber.txt", "tujuan.txt")          # copy file + permission
shutil.copy2("sumber.txt", "tujuan.txt")         # copy file + full metadata
shutil.copyfile("sumber.txt", "tujuan.txt")      # copy contents only

# Copy a directory with its contents
shutil.copytree("src_dir", "dst_dir")
shutil.copytree("src_dir", "dst_dir", dirs_exist_ok=True)  # Python 3.8+

# Move a file or directory
shutil.move("sumber.txt", "direktori_tujuan/")
shutil.move("folder_lama", "folder_baru")

# Delete a directory with all its contents
shutil.rmtree("direktori_yang_mau_dihapus")
shutil.rmtree() permanently deletes a directory and everything in it without confirmation. Make sure the path given is correct before running this function.

The json Module — Reading and Writing JSON #

JSON is the most common data exchange format. Python’s built-in json module is sufficient for most needs.

import json

data = {
    "nama": "Budi",
    "usia": 28,
    "hobi": ["membaca", "coding"],
    "aktif": True
}

# Encode Python → JSON string
json_str = json.dumps(data)
print(json_str)
# {"nama": "Budi", "usia": 28, "hobi": ["membaca", "coding"], "aktif": true}

# Pretty print
print(json.dumps(data, indent=2, ensure_ascii=False))

# Decode JSON string → Python
kembali = json.loads(json_str)
print(kembali["nama"])   # "Budi"

Reading and Writing JSON to Files #

import json

data = {"konfigurasi": {"debug": False, "port": 8080}}

# Write to a file
with open("config.json", "w", encoding="utf-8") as f:
    json.dump(data, f, indent=2, ensure_ascii=False)

# Read from a file
with open("config.json", "r", encoding="utf-8") as f:
    config = json.load(f)

print(config["konfigurasi"]["port"])   # 8080

Handling Non-Standard Data Types #

import json
from datetime import datetime

# ANTI-PATTERN: datetime can't be serialized directly
data = {"timestamp": datetime.now()}
# json.dumps(data)   # TypeError: Object of type datetime is not JSON serializable

# CORRECT: convert manually or use a custom encoder
data = {"timestamp": datetime.now().isoformat()}
print(json.dumps(data))   # {"timestamp": "2024-01-15T10:30:00.123456"}

The csv Module — Reading and Writing CSV #

The csv module handles CSV parsing correctly, including cases like values containing commas or quotes.

import csv

# Write CSV
baris_data = [
    ["nama", "usia", "kota"],
    ["Alice", 25, "Jakarta"],
    ["Bob", 30, "Bandung"],
]

with open("data.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.writer(f)
    writer.writerows(baris_data)

# Read CSV
with open("data.csv", "r", encoding="utf-8") as f:
    reader = csv.reader(f)
    for baris in reader:
        print(baris)
# ['nama', 'usia', 'kota']
# ['Alice', '25', 'Jakarta']
# ['Bob', '30', 'Bandung']

DictReader and DictWriter #

DictReader and DictWriter are more convenient because they use the header as dictionary keys, so you don’t need to remember column order.

import csv

# DictWriter -- write with column names
fieldnames = ["nama", "usia", "kota"]
with open("data.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=fieldnames)
    writer.writeheader()
    writer.writerows([
        {"nama": "Alice", "usia": 25, "kota": "Jakarta"},
        {"nama": "Bob",   "usia": 30, "kota": "Bandung"},
    ])

# DictReader -- read as dictionaries
with open("data.csv", "r", encoding="utf-8") as f:
    reader = csv.DictReader(f)
    for baris in reader:
        print(f"{baris['nama']} lives in {baris['kota']}")
# Alice lives in Jakarta
# Bob lives in Bandung
Always use newline="" when opening CSV files on Windows. Without it, every line ends with \r\r\n (double carriage return), causing blank lines between each data row.

The io Module — In-Memory IO #

The io module provides StringIO and BytesIO — objects that behave like files but live in memory. Useful for testing, processing data that doesn’t need to be saved to disk yet, or when an API expects a file object.

import io

# StringIO -- text file in memory
buffer = io.StringIO()
buffer.write("baris pertama\n")
buffer.write("baris kedua\n")
print(buffer.getvalue())
# baris pertama
# baris kedua

# Use as a file object
buffer.seek(0)   # back to the start
for baris in buffer:
    print(baris.rstrip())

# BytesIO -- binary file in memory
data_biner = io.BytesIO(b"\x89PNG\r\n\x1a\n")
print(data_biner.read(4))   # b'\x89PNG'

A real use of StringIO — testing a function that expects a file object without creating a real file:

import io
import csv

def proses_csv(file_obj):
    reader = csv.DictReader(file_obj)
    return [baris["nama"] for baris in reader]

# Use StringIO instead of a real file when testing
data_csv = "nama,usia\nAlice,25\nBob,30"
hasil = proses_csv(io.StringIO(data_csv))
print(hasil)   # ['Alice', 'Bob']

What to Use When #

Need to read/write text or binary files?
  ✓ Use open() with a with statement

Need to work with paths (navigation, existence checks, glob)?
  ✓ Use pathlib.Path  -- modern, cross-platform, recommended
  ✓ Use os.path       -- if you need old Python compatibility

Need to copy or delete a directory with its contents?
  ✓ Use shutil.copytree() / shutil.rmtree()

Need to read environment variables or process info?
  ✓ Use os.environ

Need to work with JSON data?
  ✓ Use the json module

Need to work with CSV data?
  ✓ Use csv.DictReader / csv.DictWriter

Need a file object but don't want to write to disk (testing, buffers)?
  ✓ Use io.StringIO or io.BytesIO

Summary #

  • Always use a with statement when opening files — the file is automatically closed even if an error occurs.
  • Always specify encoding="utf-8" explicitly for text files so behavior is consistent across all platforms.
  • pathlib is the modern, recommended way to work with paths — cleaner than os.path, and the / operator replaces os.path.join().
  • shutil.rmtree() deletes permanently without confirmation — make sure the path is correct before running it.
  • csv.DictReader / csv.DictWriter are better than plain csv.reader / csv.writer because you access columns by name, not index.
  • json.dumps() / json.loads() for serialization to/from strings; json.dump() / json.load() for direct to/from file.
  • io.StringIO and io.BytesIO are useful for testing and processing data in memory without touching disk.

← Previous: Strings   Next: Math →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact