Polars Python Tutorial: Fast DataFrames Without Pandas

Quick answer

Polars is a fast DataFrame library for Python, written in Rust, that handles tables of data with a small, clear API. This post loads a real orders CSV from a small shop, fixes a totals column stored as text, groups paid sales by city, and finds the average order value. If you already know Pandas, most ideas will feel familiar, and the places where Polars differs are where it earns its speed.

A small online shop in Karachi keeps its orders in a spreadsheet. Twelve rows, three cities, and a totals column that reads "Rs 8,900" instead of 8900, because that is how the shop wrote the numbers down. The owner wants three answers: which city brings in the most money, what a typical paid order is worth, and how much money sits in failed orders. A DataFrame library answers all three in a few lines. The rest of this post loads that file, trips over the totals column, fixes it, and reads the answers out.

What I would check before using it

My first check is how big the data is. Polars is written in Rust and runs operations in parallel, so it shines on files with hundreds of thousands of rows or more. A twelve-row CSV runs instantly in any library. If the tables stay small, pick the library the team already knows.

My second check is how much Pandas the team already writes. Polars looks similar, read a file, filter rows, group by a column, but the method names differ: it is group_by with an underscore, not groupby, and expressions like pl.col("total").sum() do the work instead of bare column names. The Pandas Explained: Tables in Python, Without the Spreadsheet post covers the library most teams already know.

My third check is whether the work needs lazy execution. Polars can build a query plan and run it only when asked, skipping columns and pushing filters down before reading a file. That pays off on large Parquet files. For a small CSV read in one go, eager mode is simpler and fast enough.

My fourth check is what happens after the table work. Polars hands results back as a DataFrame, and it converts cleanly to NumPy arrays and Pandas frames when another library needs them. The NumPy: Fast Arrays in Python Without the Loops post covers arrays, and the Scikit-learn Tutorial: Predict Churn With a Real Model post shows a model that takes a plain table as input.

My verdict, as judgement: adopt Polars when tables are growing past what Pandas handles comfortably, or when a new project has no Pandas code to protect. Keep Pandas when the team and its notebooks already run on it. The recommendation flips the moment a pipeline depends on a Pandas-only library with no Polars equivalent.

Twelve orders, three cities, and a totals column that will not add up

The shop's file, orders.csv, holds twelve orders from Karachi, Lahore, and Islamabad. Nine are paid and three failed. The totals look like money, "Rs 8,900", because the shop typed them the way they appear on a receipt.

order_id,customer,city,item,qty,total,status
101,Ayesha,Karachi,kurta,1,"Rs 8,900",paid
102,Bilal,Lahore,sneakers,1,"Rs 7,800",paid
103,Hina,Karachi,handbag,1,"Rs 5,400",paid
104,Usman,Islamabad,wallet,2,"Rs 6,400",paid
105,Sana,Lahore,dupatta,3,"Rs 4,500",failed
106,Omar,Karachi,shalwar kameez,1,"Rs 6,700",paid
107,Maryam,Islamabad,kurta,2,"Rs 8,900",paid
108,Ali,Karachi,sneakers,1,"Rs 9,900",failed
109,Fatima,Lahore,handbag,1,"Rs 5,400",paid
110,Hassan,Karachi,cap,4,"Rs 3,400",paid
111,Zainab,Islamabad,dupatta,2,"Rs 3,000",paid
112,Daniyal,Lahore,kurta,1,"Rs 8,900",failed

One script does the whole job: read the file, try the sum, clean the column, then group and filter.

# Shop orders with Polars for The Practical Developer (Polars demo).
#
# A small online shop in Karachi keeps its orders in a CSV file. The
# totals column holds text like "Rs 8,900", because that is how the
# shop wrote them down. This script loads the file, fixes the totals
# column, then answers three questions: how much did each city bring
# in, what is the average paid order, and how much money failed?
#
# Run from this folder:
#   python3 orders.py
import polars as pl

df = pl.read_csv("orders.csv")
print(df)
print(df.schema)

# The totals column arrived as text, so adding it up fails first.
try:
    print(df["total"].sum())
except Exception as err:
    print("ERROR:", err)

clean = df.with_columns(
    pl.col("total")
    .str.strip_prefix("Rs ")
    .str.replace_all(",", "")
    .cast(pl.Int64)
)
print(clean.schema)

paid = clean.filter(pl.col("status") == "paid")
by_city = (
    paid.group_by("city")
    .agg(
        pl.col("total").sum().alias("city_total"),
        pl.col("total").mean().round(0).alias("avg_order"),
        pl.len().alias("orders"),
    )
    .sort("city_total", descending=True)
)
print(by_city)

print("Paid total:", paid["total"].sum())
print("Paid orders:", paid.height)
print("Average paid order:", round(paid["total"].mean(), 2))

failed = clean.filter(pl.col("status") == "failed")
print("Failed total:", failed["total"].sum())
print("Failed orders:", failed.height)

Here is the real output. The table loads fine, and the schema gives the first clue: total is str, not a number.

$ python3 orders.py
shape: (12, 7)
┌──────────┬──────────┬───────────┬──────────┬─────┬──────────┬────────┐
│ order_id ┆ customer ┆ city      ┆ item     ┆ qty ┆ total    ┆ status │
│ ---      ┆ ---       ┆ ---      ┆ ---      ┆ --- ┆ ---    │
│ i64      ┆ str      ┆ str       ┆ str      ┆ i64 ┆ str      ┆ str    │
╞══════════╪══════════╪═══════════╪══════════╪═════╪══════════╪════════╡
│ 101      ┆ Ayesha   ┆ Karachi   ┆ kurta    ┆ 1   ┆ Rs 8,900 ┆ paid   │
│ 102      ┆ Bilal    ┆ Lahore    ┆ sneakers ┆ 1   ┆ Rs 7,800 ┆ paid   │
│ 103      ┆ Hina     ┆ Karachi   ┆ handbag  ┆ 1   ┆ Rs 5,400 ┆ paid   │
│ 104      ┆ Usman    ┆ Islamabad ┆ wallet   ┆ 2   ┆ Rs 6,400 ┆ paid   │
│ 105      ┆ Sana     ┆ Lahore    ┆ dupatta  ┆ 3   ┆ Rs 4,500 ┆ failed │
│ 106      ┆ Omar     ┆ Karachi   ┆ shalwar kameez ┆ 1   ┆ Rs 6,700 ┆ paid   │
│ 107      ┆ Maryam   ┆ Islamabad ┆ kurta    ┆ 2   ┆ Rs 8,900 ┆ paid   │
│ 108      ┆ Ali      ┆ Karachi   ┆ sneakers ┆ 1   ┆ Rs 9,900 ┆ failed │
│ 109      ┆ Fatima   ┆ Lahore    ┆ handbag  ┆ 1   ┆ Rs 5,400 ┆ paid   │
│ 110      ┆ Hassan   ┆ Karachi   ┆ cap      ┆ 4   ┆ Rs 3,400 ┆ paid   │
│ 111      ┆ Zainab   ┆ Islamabad ┆ dupatta  ┆ 2   ┆ Rs 3,000 ┆ paid   │
│ 112      ┆ Daniyal  ┆ Lahore    ┆ kurta    ┆ 1   ┆ Rs 8,900 ┆ failed │
└──────────┴──────────┴───────────┴──────────┴─────┴──────────┴────────┘
Schema({'order_id': Int64, 'customer': String, 'city': String, 'item': String, 'qty': Int64, 'total': String, 'status': String})
ERROR: `sum` operation not supported for dtype `str`

Hint: you may mean to call `str.join` or `list.join`
Schema({'order_id': Int64, 'customer': String, 'city': String, 'item': String, 'qty': Int64, 'total': Int64, 'status': String})

The thing that goes wrong is the totals column. Polars reads "Rs 8,900" as text, and adding up text has no meaning, so sum() raises an InvalidOperationError instead of guessing. The fix strips the "Rs " prefix and the commas, then casts to Int64, and the schema confirms the column is now a number. The same file in DuckDB: Analytics Queries on a File, Without a Server is queried with SQL instead, if the team prefers that shape.

With the column fixed, the grouping answers the shop's questions.

shape: (3, 4)
┌───────────┬────────────┬───────────┬────────┐
│ city      ┆ city_total ┆ avg_order ┆ orders │
│ ---       ┆ ---        ┆ ---       ┆ ---    │
│ str       ┆ i64        ┆ f64       ┆ u32    │
╞═══════════╪════════════╪═══════════╪════════╡
│ Karachi   ┆ 24400      ┆ 6100.0    ┆ 4      │
│ Islamabad ┆ 18300      ┆ 6100.0    ┆ 3      │
│ Lahore    ┆ 13200      ┆ 6600.0    ┆ 2      │
└───────────┴────────────┴───────────┴────────┘
Paid total: 55900
Paid orders: 9
Average paid order: 6211.11
Failed total: 23300
Failed orders: 3

Karachi leads at Rs 24,400 across 4 paid orders, then Islamabad with Rs 18,300 and Lahore with Rs 13,200, though Lahore has the highest average paid order at Rs 6,600. Nine paid orders total Rs 55,900, averaging Rs 6,211.11, while three failed orders hold Rs 23,300, money never collected. That failed total is worth chasing first: nearly a third of the money in the file never arrived.

Terminal output showing a Polars demo in Python: an orders table loading with totals stored as text, a sum failing with an InvalidOperationError, the same table grouped by city after the fix, and the paid and failed totals.


Common mistakes and how to fix them

First, typing df.groupby("city") out of Pandas habit. The symptom is an AttributeError: 'DataFrame' object has no attribute 'groupby'. The fix is group_by with an underscore, as the demo uses, followed by .agg() with the aggregations.

Second, comparing the raw totals column with a number, such as filtering for totals over 5000 before cleaning the column. The symptom is a ComputeError: cannot compare string with numeric type (i32). The fix is to clean the column first, then compare, because a text column cannot be compared with a number.

Third, casting "Rs 8,900" straight to Int64. The symptom is an InvalidOperationError: conversion from str to i64 failed in column 'total' for 12 out of 12 values. The fix is to strip the "Rs " prefix and the commas before the cast, as the demo does. Passing strict=False instead only hides the problem: every value quietly becomes null. When a fix is unclear, print df.schema first: the types usually name the problem.

Frequently Asked Questions

Is Polars a replacement for Pandas? It can be, and it is faster on large files. Pandas still has the bigger ecosystem, so a team with heavy Pandas code should switch one script at a time.

Do I need Rust to use Polars? No. Polars is written in Rust, but the Python package installs like any other with pip install polars, and the demo ran with plain Python.

Can Polars read Parquet and Excel files? Yes. It reads Parquet natively with pl.read_parquet, and Excel with pl.read_excel, which needs an extra engine installed. The CSV reader in the demo is the simplest place to start.

When should I stick with Pandas? When the project already runs on Pandas, or when a library the work depends on only accepts Pandas frames. Polars converts to Pandas with .to_pandas(), so the two can share a pipeline.

What to remember

  • Polars reads a CSV into a DataFrame, and the schema shows the type of every column.
  • A totals column stored as text must be cleaned before it can be summed.
  • group_by takes an underscore in Polars, and .agg() holds the aggregations.
  • filter() splits paid from failed orders before any total is computed.
  • Check the schema after a cast, because a silent null is worse than a loud error.

Where this leaves you

Polars earns its place the day a table stops being a toy and starts being real work: messy text where numbers should be, a question per city, and a total that has to be right. The demo proved the three things that matter: the bad totals column failed loudly instead of guessing, the fix took one expression, and the grouping gave honest answers, Rs 55,900 paid and Rs 23,300 failed. Start with one CSV and the eager API, check the schema after every cast, and reach for the lazy API when the files get big.

Further reading

Waqas Shahid is a software engineer from Karachi with more than 12 years of experience, including 6+ years in engineering leadership. He writes The Practical Developer.

Comments

Popular posts from this blog

Swift Explained: Apple's Language for iPhone, Mac, and More

LangChain Tutorial: Build Your First LLM App in Python

Rust Explained: Memory Safety Without a Garbage Collector