Finding values by position, by key, and one by one

The idea

A single price fits in a variable. A cart, a week of orders or a stock sheet needs a collection: one name that holds many values. Python has two you will use all the time, and the difference is how you find a value inside.

A list finds values by position: first, second, last. A dictionary finds them by key: "web", "marketplace", a product name. Pick the one that matches the question you ask. "What happened on day 3?" is a position. "How many orders came from the app?" is a key.

Why it exists

Imagine totalling a cart with one variable per line: line1, line2, line3. It works for three lines and falls apart at four. And you cannot write code that works for a cart of any size, because the names are fixed when you type the program. A list lets the data decide how big it is. A dictionary solves the matching problem: when stock is "Alder Oak Coffee Table: 12", scanning a list for the name each time is slow and clumsy, while a dictionary jumps straight to the answer.

Together with a loop, these are the first version of what pandas does for you later. A pandas column acts like a list, a row like a dictionary, and groupby is the counting loop below, written once and run on millions of rows.

How it works under the hood

Lists: position, slices and length

A list is ordered, and each item has a position starting at 0. len(daily) is 7, the last position is len(daily) - 1, and daily[-1] reaches it from the other end. A slice daily[start:stop] takes from start up to but not including stop, so the number of items is stop - start. Slices make copies, and slicing past the end is forgiving: daily[5:100] just returns what exists. A single position past the end is an IndexError.

Lists can be changed in place. append adds one item, and daily[0] = x replaces one. Two names pointing at the same list see the same change, which surprises people in chapter 3.

Dictionaries: keys and values

A dictionary is a set of key-value pairs. Keys are unique, and finding a value by key takes the same short time whether there are 3 keys or 3 million. That is how pandas finds a column by name, and how SQL finds a row by its primary key. d['web'] reads a key and fails with KeyError if it is not there. d.get('web', 0) reads it and falls back to 0 when it is not. Assigning d['app'] = 1 adds the key if it is new and replaces the value if it exists.

Loops and the accumulator

A for loop runs its indented block once per item, with the loop variable set to the current one. Most real loops do the same small job: they keep something that grows. That something is the accumulator, and it must exist before the loop starts.

Counting by channel is an accumulator whose value is a dictionary. A total is an accumulator whose value is a number.

The shape is always the same: create the accumulator, loop, update it from the current item, use it after the loop. If the answer is wrong, the bug is almost always one of these four: it was created inside the loop, it was updated with = instead of +=, the update is indented outside the loop, or the result is used before the loop is done.

Comprehensions

A list comprehension is a loop that builds a list, written in one line: [p for p in products if p['price'] < 20]. Read it as "p, for each p in products, if the price is under 20". It is the same loop, with the accumulator hidden. Use it when you are filtering or transforming; use a normal loop when the body needs more than one step.

Which one to reach for

Ask what you will do with the data. If the order matters and you will walk through it, slice it or take the first and last, use a list: orders by day, prices in a cart. If you will look things up by a name, or count by a label, use a dictionary: stock by product, orders by channel. When you need both, hold a list of dictionaries, one per row, with the same keys in each. That is exactly the shape of a table, and the shape pandas builds a DataFrame from.

Common misconceptions

"The first item of a list is number 1"

Positions start at 0, so daily[1] is the second day. It follows from the way a position works: it is the distance from the start, and the first item is 0 steps away. When a result is off by one, check whether you counted from 1 and Python from 0. For a slice, remember that the end is excluded: daily[0:7] is all seven days.

"A dictionary has an order like a list does"

It remembers the order keys were added in, but you can't ask for "the third entry" the way you ask a list. You ask for a key. If you need "the top channel", you loop through the pairs and keep the largest one yourself, or sort the items, which is what the levels do next.

"I can change the list I am looping over"

Adding or removing items from a list while looping over it makes the loop skip items or run forever. Build a new list instead, either with a comprehension or by appending to an empty list. The loop reads one list and writes another, and neither disturbs the other.