1. Lighthouse

Level 2 · Clean and rank the locations in pandas

The request

From Catherine Hale, CEO

“We agreed the question: consumer revenue by metro for 2025 and its growth on 2024, with the state totals next to it. Ryan warned me the addresses are a mess, so first show me how much cleaning they need.”

Your goal

Step 1 of 3. Measure the mess in customers. Set result to a one-row DataFrame with columns customer_rows (all rows), people (distinct emails after .str.strip().str.lower()), raw_states and clean_states (distinct state before and after .str.strip().str.upper()), raw_cities and clean_cities (the same for city) and missing_city (rows with a missing city).

  • Use the customers DataFramenot done yet
  • Show exactly customer_rows people raw_states clean_states raw_cities clean_cities missing_citynot done yet
  • 1 row, sorted as askednot done yet

Step 1 · your code

Python

Loading the editor…

    ResultPreview of customers · press Run to see yours
    Preview of customers
    customer_idfirst_namelast_nameemailcity
    1OwenHudsonowen.hudson@icloud.comSan Francisco
    2DouglasMendozadmendoza69@gmail.comAtlanta
    3HenryBrookshbrooks@lakesidecafe.comLos Angeles
    4BrianGrahambgraham@bluecedarhomes.comSeattle

    Mote checks your result

    Press Run. Each row that matches the goal becomes a firefly in your lantern.

    Step 2 · your code

    Python

    Loading the editor…

      ResultPreview of customers · press Run to see yours
      Preview of customers
      customer_idfirst_namelast_nameemailcity
      1OwenHudsonowen.hudson@icloud.comSan Francisco
      2DouglasMendozadmendoza69@gmail.comAtlanta
      3HenryBrookshbrooks@lakesidecafe.comLos Angeles
      4BrianGrahambgraham@bluecedarhomes.comSeattle

      Mote checks your result

      Press Run. Each row that matches the goal becomes a firefly in your lantern.

      Step 3 · your code

      Python

      Loading the editor…

        ResultPreview of customers · press Run to see yours
        Preview of customers
        customer_idfirst_namelast_nameemailcity
        1OwenHudsonowen.hudson@icloud.comSan Francisco
        2DouglasMendozadmendoza69@gmail.comAtlanta
        3HenryBrookshbrooks@lakesidecafe.comLos Angeles
        4BrianGrahambgraham@bluecedarhomes.comSeattle

        Mote checks your result

        Press Run. Each row that matches the goal becomes a firefly in your lantern.

        DataFrames

        Loaded for you as pandas DataFrames, plus pd and np.

        customers2,018 rows · 9 columns
        • customer_idinteger
        • first_namevarchar
        • last_namevarchar
        • emailvarchar
        • cityvarchar
        • statevarchar
        • signup_datedate
        • segmentvarchar
        • referral_sourcevarchar
        First 5 rows of customers
        customer_idfirst_namelast_nameemailcitystatesignup_datesegmentreferral_source
        1OwenHudsonowen.hudson@icloud.comSan FranciscoCA2022-06-02consumersearch
        2DouglasMendozadmendoza69@gmail.comAtlantaGA2022-06-02consumerNaN
        3HenryBrookshbrooks@lakesidecafe.comLos AngelesCA2022-06-04businessNaN
        4BrianGrahambgraham@bluecedarhomes.comSeattleWA2022-06-06businesspinterest
        5KyleJimenezkjimenez27@yahoo.comIndianapolisIN2022-06-07consumerNaN
        orders12,000 rows · 7 columns
        • order_idinteger
        • customer_idinteger
        • order_datetimestamp
        • statusvarchar
        • channelvarchar
        • shipping_feedecimal
        • coupon_codevarchar
        First 5 rows of orders
        order_idcustomer_idorder_datestatuschannelshipping_feecoupon_code
        10011252023-01-01 20:51:19completedweb0.00NaN
        10021562023-01-02 07:58:46completedweb0.00NaN
        1003222023-01-02 10:25:27completedmarketplace4.99WELCOME10
        1004482023-01-02 11:21:51completedmarketplace4.99WELCOME10
        1005822023-01-02 11:43:36completedweb0.00WELCOME10
        order_items30,338 rows · 6 columns
        • order_item_idinteger
        • order_idinteger
        • product_idinteger
        • quantityinteger
        • unit_pricedecimal
        • discount_pctdecimal
        First 5 rows of order_items
        order_item_idorder_idproduct_idquantityunit_pricediscount_pct
        11001121931.990.00
        21001154311.990.00
        31002154411.990.00
        410021941149.000.00
        51002193219.990.00
        products120 rows · 8 columns
        • product_idinteger
        • namevarchar
        • categoryvarchar
        • subcategoryvarchar
        • unit_pricedecimal
        • unit_costdecimal
        • launched_ondate
        • is_activeboolean
        First 5 rows of products
        product_idnamecategorysubcategoryunit_priceunit_costlaunched_onis_active
        101Vale Velvet LoveseatFurnitureSofas1659.00NaN2023-09-11true
        102Alder Oak Coffee TableFurnitureTables919.00503.502024-05-16true
        103Orchard Rattan Dining ChairFurnitureChairs789.00429.872023-05-27true
        104Upland Oak BookcaseFurnitureStorage429.00214.702022-12-20true
        105Avery Leather Bed FrameFurnitureBeds449.00232.702021-08-31true
        returns1,673 rows · 5 columns
        • return_idinteger
        • order_item_idinteger
        • returned_ondate
        • reasonvarchar
        • refund_amountdecimal
        First 5 rows of returns
        return_idorder_item_idreturned_onreasonrefund_amount
        1142023-01-10damaged83.69
        2422023-01-10wrong_item19.99
        31052023-01-14wrong_item10.99
        4242023-01-16changed_mind26.99
        51262023-01-16late20.99
        employees60 rows · 8 columns
        • employee_idinteger
        • first_namevarchar
        • last_namevarchar
        • departmentvarchar
        • titlevarchar
        • manager_idinteger
        • hire_datedate
        • salaryinteger
        First 5 rows of employees
        employee_idfirst_namelast_namedepartmenttitlemanager_idhire_datesalary
        1CatherineHaleExecutiveChief Executive OfficerNaN2016-03-01285000
        2RobertKimExecutiveChief Operating Officer12017-01-09235000
        3MeganCollinsSalesHead of Sales12018-04-16172000
        4ChloeGutierrezSalesAccount Executive32020-01-2172000
        5FrankBellSalesAccount Executive32023-05-2173000
        web_sessions40,000 rows · 7 columns
        • session_idinteger
        • customer_idinteger
        • started_attimestamp
        • devicevarchar
        • sourcevarchar
        • pages_viewedinteger
        • convertedboolean
        First 5 rows of web_sessions
        session_idcustomer_idstarted_atdevicesourcepages_viewedconverted
        1NaN2023-01-01 23:02:24mobilepaid1false
        2182023-01-01 15:07:49desktoppaid1false
        3NaN2023-01-01 16:55:46desktopdirect7false
        4NaN2023-01-01 12:43:28mobileorganic3false
        5NaN2023-01-01 13:06:21mobilesocial2false
        marketing_spend3,276 rows · 3 columns
        • spend_datedate
        • channelvarchar
        • amountdecimal
        First 5 rows of marketing_spend
        spend_datechannelamount
        2023-01-01paid492.99
        2023-01-01email37.38
        2023-01-01social200.53
        2023-01-02paid709.13
        2023-01-02email64.13

        The lesson

        What you are checking, and why

        Customer addresses are typed by customers, so the same place arrives in several spellings. Before you group by anything, measure how bad it is. nunique() on the raw column against the same count on the cleaned values tells you in two lines:

        customers["state"].nunique()                         # 69 labels
        customers["state"].str.strip().str.upper().nunique() # 39 states

        .str.strip() removes the stray spaces and .str.upper() puts every label in one case. People are matched the same way, on the cleaned email, because an email is the closest thing the table has to a person’s identity: the customer_id changes when someone signs up again.

        .str methods leave NaN as NaN and nunique() skips it, so the 14 blank cities stay out of the count instead of becoming a city called “NAN”. isna().sum() counts them; they cannot be placed in a metro and drop out of step 3.

        States first, then metros

        Step 2 gives the board the familiar state view. It mixes two sources: customers for how many people live in each state, and the order lines for who bought and how much. Each side is aggregated on its own, as a Series indexed by state, and pd.DataFrame({...}) lines them up by that index.

        Step 3 is the answer to the showroom question. It keeps consumers only, because business accounts order online and would not visit, and it ranks metros, because a showroom serves a city:

        c = customers[(customers["segment"] == "consumer") & customers["city"].notna()]
        r = lines.pivot_table(index=["city", "state"], columns="yr", values="revenue", aggfunc="sum")

        The pivot columns are the years as integers, so r[2025] - r[2024] subtracts row by row. Compute the growth on these unrounded totals and round last.

        Reading the result

        city buyers_2025 revenue_2025 growth_pct
        NEW YORK 104 96,984.37 −5.9
        LOS ANGELES 78 71,631.90 30.7
        DALLAS 58 66,362.59 17.2

        New York is the largest and Los Angeles the fastest growing of the leaders. The state table would have pointed at California as a whole, and its top-line growth of 0.3% would have hidden that Los Angeles is growing fast.

        Pitfalls

        • Cleaning a copy and then merging the raw table splits the groups again. Clean once, on customers, and merge the cleaned columns.
        • customers["city"] != customers["city"].str.strip() is True for every NaN, because NaN never equals anything. Count messy values with .notna() in the mask.
        • head(10) without sort_values returns the first ten states in alphabetical order.

        Pick what may run on this site. You can change this any time from "Cookie settings" at the bottom of every page.