How a boolean mask filters rows

The idea

A pandas filter is not a loop and not a search. It is two steps. First, a comparison on a column produces a mask: a column of True and False, one value per row. Then the square brackets keep the rows where the mask says True.

That is the whole mechanism. orders[orders['channel'] == 'marketplace'] reads awkwardly the first time because orders appears twice. The inner one builds the mask, the outer one applies it. Once you see the mask as a thing in its own right, you can name it, print it, count it, and combine it with others.

Why it exists

In plain Python you would write a loop with an if and an append. It works, and it is five lines for a one-line question. The mask does the whole table at once, like the vectorised operations in the Foundations: the comparison runs for every row in compiled code, and the result lines up with the table because it carries the same index.

It also turns filtering into something you can compose. A condition is a value. You can store it in a variable, combine two with &, flip one with ~, or average it. orders['shipping_fee'] > 0 has a mean, and that mean is the share of orders that paid for shipping, since True counts as 1 and False as 0. That trick answers every "what percentage of..." question in the boss level.

How it works under the hood

Comparison, then selection

==, !=, >, >=, < and <= compare a column with a single value or with another column of the same length, and return a mask. isin(['web', 'app']) is the mask for "is one of these", between(10, 50) for "inside a range, ends included", and str.contains('lamp') for text. Missing values compare as False, so NaN > 5 is False and the row is left out. A comparison never changes the table: only the brackets create a new one.

Combining masks

Python's and, or and not work on single True/False values, not on columns. For masks use &, | and ~, and put each condition in parentheses.

& is "and": every condition must hold, so each one narrows the result. | is "or": any condition is enough, so each one widens it. An "or" you meant as "and" is the classic error: (status == 'cancelled') | (status == 'refunded') is the lost orders, while & between them returns nothing, because no order has two statuses at once. For "one of these values" isin is clearer than a chain of |.

Labels versus positions

Every row has a label (the index) and a position (0, 1, 2 from the top). .loc selects by label and .iloc by position. On a fresh table they are the same numbers, and that is exactly why people mix them up. After a filter or a sort they drift apart: the first row of the result has position 0 and may carry the label 2. .loc also accepts a mask and column names in one go, orders.loc[mask, ['order_id', 'channel']], and unlike iloc its slices include the end label. When you want fresh labels 0, 1, 2 again, use reset_index(drop=True).

Sorting

sort_values('shipping_fee', ascending=False) reorders the rows by one column, or by several: sort_values(['channel', 'shipping_fee'], ascending=[True, False]). The later columns break ties in the earlier ones, which is what makes a "top 10" the same on every run. The labels travel with the rows, so a sorted table has an index in no particular order until you reset it. head(10) after the sort takes the top ten, and nlargest is a shorthand for that.

Build it in pieces, then check the count

A long filter is easier to trust when you build it from named masks: is_web = orders['channel'] == 'web', paid_ship = orders['shipping_fee'] > 0, then orders[is_web & paid_ship]. Each name says what it means, and you can test one at a time. Two quick checks catch most mistakes. mask.sum() counts the rows that match, because every True is a 1, and len(orders[mask]) should agree with it. If a filter returns zero rows, look at the pieces separately: one of them is usually written against a value that does not exist, like 'Web' for 'web'.

Filter first, then ask the question

Filtering is rarely the answer. It is the step before one. "Average basket of completed web orders" is a filter followed by a mean. "Which marketplace orders paid the highest fee" is a filter followed by a sort and head. Keeping the two apart, mask first and question second, lets you check the filtered table by eye before the number you report is computed from it. If the filtered table looks wrong, the number will be wrong however clean the last line is.

Common misconceptions

"Filtering changes the DataFrame"

It does not. orders[mask] returns a new DataFrame and leaves orders as it was. If you want to keep the filtered table, assign it: marketplace = orders[mask]. This also explains why a filter you ran an hour ago did "nothing" to your original. You looked at a result and never saved it.

"I can write a condition with and / or"

orders['a'] > 1 and orders['b'] < 5 raises "the truth value of a Series is ambiguous", because Python asks "is this whole column True?" and a column has many values. Use & and | with parentheses. The error text sounds deep and the cause is just the operator.

"The first row of a result is row 0"

Only on a fresh table. After a filter it is the first row that survived, with its original label, so result.loc[0] may fail with a KeyError while result.iloc[0] works. If you only need the rows and not the old labels, call reset_index(drop=True) and the two agree again.