# Python Write to File Line by Line

![Python Write to File Line by Line](https://raw.githubusercontent.com/mateenkiani/milddev-resources/main/images/1753990413737_python-write-to-file-line-by-line.webp)

# Python Write to File Line by Line

## Introduction
Writing data to files is a core task for any Python developer. Whether you're logging events, generating reports, or storing user input, writing one line at a time gives you control and readability. In this article, we'll walk through multiple approaches to write to files line by line in Python.

Understanding how to write incrementally can help you avoid memory spikes, handle large datasets, and make your logs more organized. Ready to see how small changes in your code can make file I/O smoother and more reliable?

By mastering these simple techniques, you'll write cleaner, more efficient code and prevent common pitfalls like file corruption or unexpected overwrites. Let's dive in!

## Why Write Line by Line
Writing line by line lets you:

- Stream large datasets without loading everything in memory
- Interleave processing and output for real-time logging
- Append new entries without rewriting the entire file

> Tip: When dealing with huge files, writing one line at a time keeps memory usage low and avoids `MemoryError`.

Python’s built-in file object makes this straightforward. You’ll see two main patterns: using a loop with `write()` and leveraging `writelines()` for lists.

## Using a Loop with write()
The most explicit way is to open a file and call `write()` for each line:

```python
lines = [
    "First line of output\n",
    "Second line of output\n",
    "Third line of output\n"
]

with open('output.txt', 'w', encoding='utf-8') as f:
    for line in lines:
        f.write(line)
```

Key points:

- The `with` statement ensures the file closes automatically, even on errors.
- Always include a newline `\n` at the end of each line.
- Choose the right mode: `'w'` for overwrite, `'a'` to append.

### Append Mode
To add new lines without erasing existing data:

```python
with open('output.txt', 'a', encoding='utf-8') as f:
    f.write("Another line added later\n")
```

## Writing from an Iterable with writelines()
If you already have a list or generator of lines, `writelines()` is concise:

```python
log_entries = (f"Entry {i}: data processed\n" for i in range(1000))

with open('log.txt', 'w') as logfile:
    logfile.writelines(log_entries)
```

Note:

- `writelines()` does **not** add newlines automatically. Each element must end with `\n`.
- It writes all lines in one go, which can be faster but uses more memory.

> Tip: If you need to save a Python list, see our guide on [saving a Python list to a file](https://milddev.com/how-to-save-a-python-list-to-a-file).

## Flushing and Buffering
Python buffers writes for performance. In most cases, you don’t need to worry about it. But for real-time logs or progress updates:

```python
with open('progress.txt', 'w') as f:
    for i in range(5):
        f.write(f"Step {i}\n")
        f.flush()  # Force write to disk
        time.sleep(1)
```

> Warning: Excessive flushing can slow down your program.

## Handling File Encodings
Different platforms use different default encodings. To avoid surprises:

```python
with open('data.txt', 'w', encoding='utf-8', errors='replace') as f:
    f.write("Some unicode text: café, 北京\n")
```

- `encoding='utf-8'` ensures consistent behavior across systems
- `errors='replace'` or `'ignore'` handles unexpected byte sequences

## Serializing Complex Data
When you need structured formats like JSON, write line by line as serialized strings:

```python
import json
records = [
    {'id': 1, 'status': 'ok'},
    {'id': 2, 'status': 'error'}
]

with open('records.jsonl', 'w') as f:
    for rec in records:
        line = json.dumps(rec)
        f.write(line + "\n")
```

> Pro tip: JSON Lines (`.jsonl`) files are great for streaming large logs.

For more advanced JSON file writing, check out our [writing JSON to files in Python](https://milddev.com/python-write-json-to-file) guide.

## Common Pitfalls and Best Practices

- **Missing newline**: Forgetting `\n` results in mashed lines.
- **Wrong mode**: Using `'w'` instead of `'a'` overwrites existing data.
- **No exception handling**: Wrap critical writes in `try/except` if failure is possible.
- **Hard-coded paths**: Use `os.path` or `pathlib` to build file paths.

**Best Practices:**

1. Always use `with open(...)`.
2. Specify `encoding` explicitly.
3. Choose the smallest buffer you need.
4. Flush sparingly.

## Conclusion
Line-by-line writing in Python is simple yet powerful. You can stream large data, create real-time logs, or serialize complex records without loading everything into memory. By using `write()`, `writelines()`, and careful buffering, your applications will run efficiently and predictably. Remember to handle encodings explicitly and choose the correct file mode to avoid data loss.

Next time you need to record output or logs, lean on these patterns to keep your code clean and your files intact. Happy coding!