The idea
You know from the Foundations that a DataFrame is a table of typed columns. This chapter is about the habit that goes with it: before you calculate anything, look. A handful of one-line calls tell you how big the table is, what each column holds, which values show up and how often. Skipping them is how a total gets computed on the wrong column, or on a table with duplicated rows, and then sent to a VP.
Figure 1
One bracket gives a Series, two brackets give a DataFrame
| order_id | status | channel | shipping_fee |
|---|---|---|---|
| 1001 | completed | web | 0.00 |
| 1002 | completed | web | 0.00 |
| 1003 | completed | marketplace | 4.99 |
| 1006 | cancelled | marketplace | 4.99 |
| status |
|---|
| completed |
| completed |
| completed |
| cancelled |
| order_id | status |
|---|---|
| 1001 | completed |
| 1002 | completed |
| 1003 | completed |
| 1006 | cancelled |
- Start from
orders, a DataFrame with four columns. You almost never need all of them for one question. orders['status'], one name in single brackets, returns a Series: one column of values, with the row labels attached. It has methods for values, likevalue_counts().orders[['order_id', 'status']], a list inside the brackets, returns a DataFrame with just those columns, still a table. The inner brackets are a Python list of names.
The first thing to get right is what you are holding. A DataFrame is the whole table. A Series is one column of it. Single brackets with one name give you a Series, double brackets with a list give you a DataFrame. They print almost the same, but they have different methods, and mixing them up is the commonest error message of the chapter.
Why it exists
Real tables are never as clean as you were told. A column documented as "channel" has a value nobody mentioned. A "date" is stored as text. A key you were sure was unique repeats. These problems cost nothing to find at the start and a great deal to find after the report has gone out. The first-look calls take seconds, and their value is that you ask the same questions of every new table, in the same order, without having to think about it.
How it works under the hood
The size and the types
orders.shape gives (12000, 7): rows first, columns second. len(orders) gives 12,000, the number of rows. orders.info() lists every column with its dtype and its count of non-missing values, so one glance tells you which columns are numbers, which are text, and which have holes. orders.head() shows the first five rows, so you can read a few real values next to the column names.
Counting the values
For a column of categories, value_counts() is the first call to make. It returns a new Series: the distinct values as its index, and the counts as its values, biggest first.
Figure 2
orders['status'].value_counts(): how the 12,000 orders ended
value_counts()counts how often each distinct value appears, biggest first. Of 12,000 orders, 10,849 arecompleted: about 9 in 10.- The rest is a short tail. 726 cancelled and 304 refunded are the two you would investigate; 121 are still pending.
- Four values, four bars, and the counts add up to 12,000. A fifth value, a typo like
Completed, would show here as its own bar. That is what a first look is for.
With normalize=True it gives fractions instead of counts, which is how you say "90% of orders completed". For a numeric column, describe() gives the count, mean, standard deviation, minimum, quartiles and maximum in one table. Read it for sanity: a minimum price of 0, or a maximum 100 times the median, is worth a look before any average is trusted.
Counting rows versus counting things
Three calls look alike and answer different questions. len(df) is the number of rows. series.count() is the number of rows with a value, so it skips missing ones. series.nunique() is the number of different values.
Figure 3
count() and nunique() answer different questions
| order_id | customer_id |
|---|---|
| 1001 | 125 |
| 1002 | 156 |
| 1003 | 22 |
| 1004 | 48 |
| 1005 | 82 |
| 1006 | 157 |
| 1007 | 21 |
| 1008 | 157 |
| 1009 | 38 |
| 1010 | 111 |
| call | result |
|---|---|
| len(orders) | 10 |
| customer_id.count() | 10 |
| customer_id.nunique() | 9 |
- Ten orders. Each has a
customer_id, but customers can order more than once. count()counts the values that are not missing. All ten orders have a customer, so it is 10, the same aslen.- Customer 157 placed orders 1006 and 1008.
nunique()counts distinct values, so 157 counts once, and the answer is 9 customers.
The second and third are where wrong answers hide. "How many customers ordered?" is nunique() on customer_id; the row count would count every repeat purchase. "How many orders are missing a coupon?" is isna().sum() on coupon_code, and count() is its opposite. unique() gives the distinct values themselves, not just how many, which is the quickest way to spot web and Web in the same column.
Loading a file
pd.read_csv reads a file into a DataFrame and guesses each column's dtype from what it sees. Dates arrive as text unless you say parse_dates=['order_date']. A numeric column with one stray "n/a" arrives as text. So after loading, dtypes is the first check, before any arithmetic. NumPy, which pandas is built on, supplies the arrays under the columns, which is why np.mean, cumsum and element-wise arithmetic work on them directly.
A routine you can repeat
Run the same short routine on every new table. First shape and head(): how much is there and what does a row look like? Then info(): are the types right and where are the gaps? Then value_counts() on each category column and describe() on the numeric ones: do the values and the ranges make sense? Last, nunique() on every column that should be a key: is it really one per row? Write down anything odd, such as a status you did not expect, a price of 0 or a date stored as text. Five minutes here usually saves an hour of confused debugging later, and it gives you a short list of questions to ask whoever owns the data.
Common misconceptions
"A Series and a DataFrame with one column are the same"
They look the same when printed and behave differently. A Series has no columns to select from, so s[['a', 'b']] means something else entirely; a one-column DataFrame is still a table you can keep working on. If a method complains about "no attribute", check which of the two you hold with type(x), and use double brackets whenever you want to keep a table.
"count() tells me how many rows there are"
count() counts values that are present. On a column with missing data it is smaller than the row count, and the gap is exactly the number of blanks. For "how many rows" use len or shape[0]. For "how many different" use nunique(). Picking the wrong one gives a plausible number, which is worse than an error.
"If the data loaded without errors it is fine"
pandas will happily load a column of dates as text, a price with a currency sign as text, and a key with duplicates. None of these raise an error. A clean load means the file was readable, not that the data is right. The first look is how you find out the difference.
Check yourself
One question. Answering it ticks this lesson off on the map; it never locks a level.
Not quite. That is backwards. len(orders) counts rows, which are orders, and nunique() counts distinct customers.Try another answer.
Right. len counts every order row, while nunique() counts each customer once however many orders they placed. Fewer customers than orders means repeat buyers.
Not quite. Missing values are what count() and isna() reveal. nunique() counts distinct values and says nothing about missing ones.Try another answer.
Not quite. They answer different questions, so they only match when every customer ordered exactly once.Try another answer.
Lesson done: it is ticked on the map.