Web Scraping

Lukas Hager

2024-11-28

Learning Objectives

Understand the appropriate use of web scraping as a tool
Know how to use the requests module to access a page’s HTML
Use BeautifulSoup to parse HTML
Learn to inspect pages and understand how to best extract data

Ethics

Is Web Scraping Legal?

Ignorantia juris non excusat.

Probably

Is Web Scraping Ethical?

That depends
- Is the data that you’re scraping publicly available?
- Are you scraping to sell the data?
- Are you scraping to make money off of using the data otherwise?
- Are you disrupting use of the site by others?
Check out robots.txt file if it exists
- Guide for Google’s and Yahoo’s crawlers
- Newly, also for ChatGPT and other LLMs

Basics

What is Web Scraping?

Acquiring data from a website programmatically
Imagine copy-pasting tabular data from webpages, but using code
Very useful for creating datasets that are directly of interest

Basic Goal

We want to ingest HTML from a page and return a dataframe of the data that we want from that page

`requests`

Python has a built-in module for accessing webpages
Here, we’re going to access Yahoo’s webpage of Apple’s stock price history
- This is hosted on my website so we don’t get in trouble

import requests

req_obj = requests.get(
    'https://lukashager.netlify.app/econ-481/data/yahoo_apple_data.html'
)

`status_code`

A status_code of 200 means that your request was successful

req_obj.status_code

`ok`

You can more directly make sure the request was successful with the ok attribute

req_obj.ok

True

Content

What did requests actually give us? The HTML of the site we requested:

req_obj.text[:2000]

'<!DOCTYPE html>\n<!-- saved from url=(0045)https://finance.yahoo.com/quote/AAPL/history/ -->\n<html lang="en-US" theme="light" data-color-scheme="light" class="desktop neo-green ybar-page-is-scrolled"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8">\n        \n        <meta name="viewport" content="width=device-width, initial-scale=1">\n        <meta name="oath:guce:consent-host" content="guce.yahoo.com">\n        <link rel="manifest" href="https://finance.yahoo.com/manifest.json">\n        \n\t\t<link href="./Apple Inc. (AAPL) Stock Historical Prices &amp; Data - Yahoo Finance_files/2.BTQWGhbx.css" rel="stylesheet">\n\t\t<link href="./Apple Inc. (AAPL) Stock Historical Prices &amp; Data - Yahoo Finance_files/Debug.Be8r5cPM.css" rel="stylesheet">\n\t\t<link href="./Apple Inc. (AAPL) Stock Historical Prices &amp; Data - Yahoo Finance_files/Button.wzorZdtC.css" rel="stylesheet">\n\t\t<link href="./Apple Inc. (AAPL) Stock Historical Prices &amp; Data - Yahoo Finance_files/Icon.5Pcv_1eF.css" rel="stylesheet">\n\t\t<link href="./Apple Inc. (AAPL) Stock Historical Prices &amp; Data - Yahoo Finance_files/Tooltip._ruFWoPl.css" rel="stylesheet">\n\t\t<link href="./Apple Inc. (AAPL) Stock Historical Prices &amp; Data - Yahoo Finance_files/AccordionItem.qVASFip0.css" rel="stylesheet">\n\t\t<link href="./Apple Inc. (AAPL) Stock Historical Prices &amp; Data - Yahoo Finance_files/Link.zOM28wTY.css" rel="stylesheet">\n\t\t<link href="./Apple Inc. (AAPL) Stock Historical Prices &amp; Data - Yahoo Finance_files/Switch.C38o_CYo.css" rel="stylesheet">\n\t\t<link href="./Apple Inc. (AAPL) Stock Historical Prices &amp; Data - Yahoo Finance_files/Tabs.WO-8j5ao.css" rel="stylesheet">\n\t\t<link href="./Apple Inc. (AAPL) Stock Historical Prices &amp; Data - Yahoo Finance_files/70.CythSXdU.css" rel="stylesheet">\n\t\t<link href="./Apple Inc. (AAPL) Stock Historical Prices &amp; Data - Yahoo Finance_files/EventCalendarPreview.BSw8eqA0.css" rel="stylesheet">\n\t\t<link href="./Apple Inc. (AAPL) Stock Historica'

HTML

If we request a webpage, we’re going to get back the page’s HTML code
If the site is very simple, the HTML generates the full output (imagine really old MS Paint websites)
Now, normally there are scripts that also run within the page and render objects or request additional data – can make things tricky

HTML Structure

A very basic piece of HTML might look like this:

<div>
    <h4> This is a heading </h4>
    <p> This is a sentence </p>
</div>

This is a heading

This is a sentence

HTML Tables

We often care about tables when web scraping – they look like this:

<table>
    <thead>
        <tr> 
            <td> a column </td>
            <td> another column </td>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td> 1 </td>
            <td> 2 </td>
        </tr>
    </tbody>
</table>

a column	another column
1	2

`BeautifulSoup`

A library that we can use to process HTML more easily
Facilitates searching for specific elements in a webpage

`prettify()`

We can use BeautifulSoup to make the page’s HTML more nicely formatted and readable – we won’t actually run the command here since the output is quite large.

from bs4 import BeautifulSoup

soup = BeautifulSoup(req_obj.text)
# print(soup.prettify()) this would print formatted HTML

Searching in the HTML

We only want some of the HTML – how should we figure out what we want and where it is?
Easiest way: using Chrome, press Ctrl + Shft + C on PC or Cmd + Shft + C on Mac
Alternatively (still in Chrome) right click on the element you care about and click “Inspect”

Finding `table` in `BeautifulSoup`

BeautifulSoup allows us to search for specific HTML tag by name – if there’s only one table in the page, using find with the tag’s name should work well

table_obj = soup.find('table')

Making this Usable

We have a thead tag
- Represents the headers of the table
We then have tbody tag
- Represents the table content

Getting the Headers

table_obj.find('thead').find_all('th')

[<th class="svelte-ta1t6m">Date  </th>,
 <th class="svelte-ta1t6m">Open  </th>,
 <th class="svelte-ta1t6m">High  </th>,
 <th class="svelte-ta1t6m">Low  </th>,
 <th class="svelte-ta1t6m">Close <span class="container svelte-ur1qpn"><div class="icon fin-icon primary-icn sz-medium tw-align-text-top svelte-21xhfv"><svg viewbox="0 0 24 24" xmlns="http://www.w3.org/2000/svg"><path d="M11 7h2v2h-2zm0 4h2v6h-2zm1-9C6.48 2 2 6.48 2 12s4.48 10 10 10 10-4.48 10-10S17.52 2 12 2m0 18c-4.41 0-8-3.59-8-8s3.59-8 8-8 8 3.59 8 8-3.59 8-8 8"></path></svg></div></span> <div class="tooltip al-bottom svelte-ur1qpn" role="tooltip"><div class="arrow svelte-ur1qpn"></div> <h3 class="title svelte-ur1qpn"></h3> <span class="toolTipContent svelte-ta1t6m" slot="content">Close price adjusted for splits.</span></div> </th>,
 <th class="svelte-ta1t6m">Adj Close <span class="container svelte-ur1qpn"><div class="icon fin-icon primary-icn sz-medium tw-align-text-top svelte-21xhfv"><svg viewbox="0 0 24 24" xmlns="http://www.w3.org/2000/svg"><path d="M11 7h2v2h-2zm0 4h2v6h-2zm1-9C6.48 2 2 6.48 2 12s4.48 10 10 10 10-4.48 10-10S17.52 2 12 2m0 18c-4.41 0-8-3.59-8-8s3.59-8 8-8 8 3.59 8 8-3.59 8-8 8"></path></svg></div></span> <div class="tooltip al-bottom svelte-ur1qpn" role="tooltip"><div class="arrow svelte-ur1qpn"></div> <h3 class="title svelte-ur1qpn"></h3> <span class="toolTipContent svelte-ta1t6m" slot="content">Adjusted close price adjusted for splits and dividend and/or capital gain distributions.</span></div> </th>,
 <th class="svelte-ta1t6m">Volume  </th>]

Converting `BeautifulSoup` to List

headers = [x.text for x in table_obj.find('thead').find_all('th')]
headers

['Date  ',
 'Open  ',
 'High  ',
 'Low  ',
 'Close    Close price adjusted for splits. ',
 'Adj Close    Adjusted close price adjusted for splits and dividend and/or capital gain distributions. ',
 'Volume  ']

Ugly, but usable!

Making Headers Prettier

import re
headers_pretty = [
    re.findall('[A-Za-z]+\s?[A-Za-z]+(?=\s+)', x)[0] for x in headers
]
headers_pretty

['Date', 'Open', 'High', 'Low', 'Close', 'Adj Close', 'Volume']

Exercise: Getting Table Contents

Use a similar approach to get the contents of the table using BeautifulSoup. Hint: the HTML tag for the body is tbody, the tag for rows is tr, and the tag for elements is td.

Solutions: Getting Table Contents

rows = table_obj.find('tbody').find_all('tr')
data_list = [[item.text for item in row.find_all('td')] for row in rows]
data_list[:2]

[['Mar 25, 2024',
  '170.53',
  '171.94',
  '169.45',
  '170.85',
  '170.85',
  '53,895,981'],
 ['Mar 22, 2024',
  '171.76',
  '173.05',
  '170.06',
  '172.28',
  '172.28',
  '71,106,600']]

Putting It Together

import pandas as pd

df = pd.DataFrame(
    data = data_list,
    columns = headers_pretty
).set_index('Date')
print(df.head(10))

                Open    High     Low   Close Adj Close       Volume
Date                                                               
Mar 25, 2024  170.53  171.94  169.45  170.85    170.85   53,895,981
Mar 22, 2024  171.76  173.05  170.06  172.28    172.28   71,106,600
Mar 21, 2024  177.05  177.49  170.84  171.37    171.37  106,181,300
Mar 20, 2024  175.72  178.67  175.09  178.67    178.67   53,423,100
Mar 19, 2024  174.34  176.61  173.03  176.08    176.08   55,215,200
Mar 18, 2024  175.57  177.71  173.52  173.72    173.72   75,604,200
Mar 15, 2024  171.17  172.62  170.29  172.62    172.62  121,664,700
Mar 14, 2024  172.91  174.31  172.05  173.00    173.00   72,913,500
Mar 13, 2024  172.77  173.19  170.76  171.13    171.13   52,488,700
Mar 12, 2024  173.15  174.03  171.01  173.23    173.23   59,825,400

Faster Solution (If Possible)

print(pd.read_html(req_obj.text)[0].head(10))

           Date    Open    High     Low  \
0  Mar 25, 2024  170.53  171.94  169.45   
1  Mar 22, 2024  171.76  173.05  170.06   
2  Mar 21, 2024  177.05  177.49  170.84   
3  Mar 20, 2024  175.72  178.67  175.09   
4  Mar 19, 2024  174.34  176.61  173.03   
5  Mar 18, 2024  175.57  177.71  173.52   
6  Mar 15, 2024  171.17  172.62  170.29   
7  Mar 14, 2024  172.91  174.31  172.05   
8  Mar 13, 2024  172.77  173.19  170.76   
9  Mar 12, 2024  173.15  174.03  171.01   

  Close Close price adjusted for splits.  \
0                                 170.85   
1                                 172.28   
2                                 171.37   
3                                 178.67   
4                                 176.08   
5                                 173.72   
6                                 172.62   
7                                 173.00   
8                                 171.13   
9                                 173.23   

  Adj Close Adjusted close price adjusted for splits and dividend and/or capital gain distributions.  \
0                                             170.85                                                   
1                                             172.28                                                   
2                                             171.37                                                   
3                                             178.67                                                   
4                                             176.08                                                   
5                                             173.72                                                   
6                                             172.62                                                   
7                                             173.00                                                   
8                                             171.13                                                   
9                                             173.23                                                   

      Volume  
0   53895981  
1   71106600  
2  106181300  
3   53423100  
4   55215200  
5   75604200  
6  121664700  
7   72913500  
8   52488700  
9   59825400

When Does `pd.read_html` Work?

If you have a simple <table> tag, pd.read_html will probably work
Note that the column names are still ugly – pd.read_html will follow our approach above
Note that in this specific application, there are plenty of sites that allow direct CSV download of stock data, so scraping is superfluous

Harder Scraping Problem

Baseball Reference

gunnar = requests.get('https://www.baseball-reference.com/players/h/hendegu01.shtml')
gunnar.status_code

Let’s get the “Advanced Batting” table

Find “Advanced Batting”

There are a lot of tables on the page

gunnar_bs = BeautifulSoup(gunnar.text)
len(gunnar_bs.find_all('table'))

Huh?

Use `pandas`

pd.read_html(gunnar.text)

[         Season           Age          Team            Lg   WAR    G    PA  \
 0          2022            21           BAL            AL   1.0   34   132   
 1          2023            22           BAL            AL   6.2  150   622   
 2          2024            23           BAL            AL   9.1  159   719   
 3         3 Yrs         3 Yrs         3 Yrs         3 Yrs  16.2  343  1473   
 4  162 Game Avg  162 Game Avg  162 Game Avg  162 Game Avg   7.7  162   696   
 
      AB    R    H  ...   rOBA  Rbat+   TB  GIDP  HBP  SH  SF  IBB     Pos  \
 0   116   12   30  ...  0.348    128   51     1    0   0   0    1  5/6DH4   
 1   560  100  143  ...  0.349    125  274    10    3   0   3    1   56D/H   
 2   630  118  177  ...  0.385    157  333     2    7   0   4    1    *6/D   
 3  1306  230  350  ...  0.367    140  658    13   10   0   7    3  65DH/4   
 4   617  109  165  ...  0.367    140  311     6    5   0   3    1     NaN   
 
            Awards  
 0             NaN  
 1  MVP-8,ROY-1,SS  
 2        AS,MVP-4  
 3             NaN  
 4             NaN  
 
 [5 rows x 33 columns],
          Season           Age          Team            Lg    G   PA   AB   R  \
 0          2023            22           BAL            AL    3   13   12   3   
 1          2024            23           BAL            AL    2    9    7   0   
 2         2 Yrs         2 Yrs         2 Yrs         2 Yrs    5   22   19   3   
 3  162 Game Avg  162 Game Avg  162 Game Avg  162 Game Avg  162  713  616  97   
 
      H  2B  ...    SLG    OPS   TB  GIDP  HBP  SH  SF  IBB   Pos  Awards  
 0    6   0  ...  0.750  1.288    9     0    0   0   0    0  /*56     NaN  
 1    0   0  ...  0.000  0.222    0     0    0   0   0    0   /*6     NaN  
 2    6   0  ...  0.474  0.883    9     0    0   0   0    0  /*65     NaN  
 3  194   0  ...  0.474  0.883  292     0    0   0   0    0   NaN     NaN  
 
 [4 rows x 29 columns],
          Season           Age          Team            Lg    PA  Rbat  Rbaser  \
 0          2022            21           BAL            AL   132     4      -1   
 1          2023            22           BAL            AL   622    18       3   
 2          2024            23           BAL            AL   719    46       3   
 3         3 Yrs         3 Yrs         3 Yrs         3 Yrs  1473    68       6   
 4  162 Game Avg  162 Game Avg  162 Game Avg  162 Game Avg   696    32       3   
 
    Rdp  Rfield  Rpos  ...  Rrep  RAR   WAR  waaWL%  162WL%  oWAR  dWAR  oRAR  \
 0    1       0     1  ...     5   10   1.0   0.516   0.503   1.0   0.1    10   
 1    0      13     6  ...    21   62   6.2   0.527   0.525   4.8   2.0    49   
 2    2       5    10  ...    23   89   9.1   0.543   0.542   8.5   1.5    84   
 3    4      18    16  ...    49  161  16.2   0.533   0.531  14.3   3.6   143   
 4    2       9     8  ...    23   76   7.7   0.533   0.531   6.8   1.7    67   
 
       Pos          Awards  
 0  5/6DH4             NaN  
 1   56D/H  MVP-8,ROY-1,SS  
 2    *6/D        AS,MVP-4  
 3  65DH/4             NaN  
 4     NaN             NaN  
 
 [5 rows x 22 columns],
    Unnamed: 0_level_0 Unnamed: 1_level_0 Unnamed: 2_level_0  \
                Season                Age               Team   
 0                2022                 21                BAL   
 1                2022                 21                BAL   
 2                2022                 21                BAL   
 3                2023                 22                BAL   
 4                2023                 22                BAL   
 5                2024                 23                BAL   
 6               3 Yrs              3 Yrs              3 Yrs   
 7                 NaN                NaN                NaN   
 8          SS (3 Yrs)         SS (3 Yrs)         SS (3 Yrs)   
 9          3B (2 Yrs)         3B (2 Yrs)         3B (2 Yrs)   
 10          2B (1 Yr)          2B (1 Yr)          2B (1 Yr)   
 
    Unnamed: 3_level_0 Unnamed: 4_level_0 Standard                        \
                    Lg                Pos        G     GS     CG     Inn   
 0                  AL                 3B     24.0   18.0   17.0   172.2   
 1                  AL                 SS      7.0    6.0    4.0    48.0   
 2                  AL                 2B      3.0    3.0    2.0    26.0   
 3                  AL                 3B     84.0   68.0   47.0   594.2   
 4                  AL                 SS     83.0   64.0   50.0   584.2   
 5                  AL                 SS    157.0  157.0  149.0  1384.0   
 6               3 Yrs                NaN    358.0  316.0  269.0  2810.0   
 7                 NaN                NaN      NaN    NaN    NaN     NaN   
 8          SS (3 Yrs)                 SS    247.0  227.0  203.0  2016.2   
 9          3B (2 Yrs)                 3B    108.0   86.0   64.0   767.1   
 10          2B (1 Yr)                 2B      3.0    3.0    2.0    26.0   
 
             ...        Total Zone           DRS         Range Factor        \
         Ch  ... lgFld%       Rtot Rtot/yr  Rdrs Rdrs/yr         RF/9 lgRF9   
 0     57.0  ...  0.967        3.0    19.0   1.0     7.0         2.97  2.70   
 1     33.0  ...  0.972        2.0    53.0   1.0    25.0         6.00  3.92   
 2     16.0  ...  0.982       -1.0   -37.0  -2.0   -92.0         5.54  4.15   
 3    159.0  ...  0.962        1.0     1.0   3.0     6.0         2.32  2.49   
 4    259.0  ...  0.972       10.0    19.0  10.0    21.0         3.86  3.68   
 5    635.0  ...  0.972        1.0     1.0   5.0     4.0         3.97  3.70   
 6   1159.0  ...  0.971       15.0     6.0  18.0     8.0         3.58  3.39   
 7      NaN  ...    NaN        NaN     NaN   NaN     NaN          NaN   NaN   
 8    927.0  ...  0.972       12.0     7.0  16.0    10.0         3.99  3.70   
 9    216.0  ...  0.963        3.0     5.0   4.0     6.0         2.46  2.54   
 10    16.0  ...  0.982       -1.0   -37.0  -2.0   -92.0         5.54  4.15   
 
                Unnamed: 24_level_0  
     RF/G lgRFG              Awards  
 0   2.38  2.67                 NaN  
 1   4.57  3.86                 NaN  
 2   5.33  4.10                 NaN  
 3   1.82  2.46      MVP-8,ROY-1,SS  
 4   3.02  3.63      MVP-8,ROY-1,SS  
 5   3.89  3.65            AS,MVP-4  
 6   3.13  3.31                 NaN  
 7    NaN   NaN                 NaN  
 8   3.62  3.65                 NaN  
 9   1.94  2.51                 NaN  
 10  5.33  4.10                 NaN  
 
 [11 rows x 25 columns],
   Unnamed: 0_level_0 Unnamed: 1_level_0 Unnamed: 2_level_0 Unnamed: 3_level_0  \
               Season                Age               Team                 Lg   
 0               2023                 22                BAL                 AL   
 1               2023                 22                BAL                 AL   
 2               2024                 23                BAL                 AL   
 3              2 Yrs              2 Yrs              2 Yrs              2 Yrs   
 4                NaN                NaN                NaN                NaN   
 5         SS (2 Yrs)         SS (2 Yrs)         SS (2 Yrs)         SS (2 Yrs)   
 6          3B (1 Yr)          3B (1 Yr)          3B (1 Yr)          3B (1 Yr)   
 
   Unnamed: 4_level_0 Standard                                            \
                  Pos        G   GS   Inn    Ch   PO    A    E   DP Fld%   
 0                 3B      3.0  1.0  12.0   3.0  1.0  2.0  0.0  1.0  1.0   
 1                 SS      2.0  2.0  14.0   4.0  2.0  2.0  0.0  0.0  1.0   
 2                 SS      2.0  2.0  18.0   7.0  3.0  4.0  0.0  0.0  1.0   
 3                NaN      7.0  5.0  44.0  14.0  6.0  8.0  0.0  1.0  1.0   
 4                NaN      NaN  NaN   NaN   NaN  NaN  NaN  NaN  NaN  NaN   
 5                 SS      4.0  4.0  32.0  11.0  5.0  6.0  0.0  0.0  1.0   
 6                 3B      3.0  1.0  12.0   3.0  1.0  2.0  0.0  1.0  1.0   
 
   Range Factor       Unnamed: 16_level_0  
           RF/9  RF/G              Awards  
 0         2.25  1.00                 NaN  
 1         2.57  2.00                 NaN  
 2         3.50  3.50                 NaN  
 3         2.86  2.00                 NaN  
 4          NaN   NaN                 NaN  
 5         3.09  2.75                 NaN  
 6         2.25  1.00                 NaN  ]

`id` tag

gunnar_bs.find_all('div', {'id': 'div_batting_advanced'})

[]

Huh?

Search the Raw HTML

re.findall('Advanced Batting', gunnar.text)

['Advanced Batting',
 'Advanced Batting',
 'Advanced Batting',
 'Advanced Batting',
 'Advanced Batting']

So in the HTML there is an Advanced Batting table somewhere – it’s just being stored in a bizarre format.
At this point, best to just inspect the raw HTML

Getting Tables in Comments

We know that tables are sneakily hidden in comments – how can we get around this?

all_tables = re.findall(
    '\<table.*?\</table\>', 
    gunnar.text, 
    flags=re.DOTALL
)
len(all_tables)

Get The Table We Care About

adv_batting = [x for x in all_tables if 'batting_advanced' in x]
len(adv_batting)

Use `pd.read_html`

from io import StringIO # used to wrap raw text passed to pandas

pd.read_html(StringIO(adv_batting[0]))

[  Unnamed: 0_level_0 Unnamed: 1_level_0 Unnamed: 2_level_0 Unnamed: 3_level_0  \
                 Year                Age                 Tm                 Lg   
 0               2022                 21                BAL                 AL   
 1               2023                 22                BAL                 AL   
 2               2024                 23                BAL                 AL   
 3              3 Yrs              3 Yrs              3 Yrs              3 Yrs   
 4       MLB Averages       MLB Averages       MLB Averages       MLB Averages   
 
   Batting                     Batting Ratios         ... Batted Ball         \
      rOBA Rbat+  BAbip    ISO            HR%    SO%  ...       GB/FB  Pull%   
 0   0.348   128  0.333  0.181           3.0%  25.8%  ...        1.34  30.5%   
 1   0.349   125  0.306  0.234           4.5%  25.6%  ...        0.84  34.2%   
 2   0.385   157  0.320  0.248           5.2%  22.1%  ...        0.90  32.2%   
 3   0.367   140  0.315  0.236           4.7%  23.9%  ...        0.91  32.9%   
 4   0.319   100  0.292  0.158           3.0%  22.6%  ...        0.74  30.0%   
 
                 Win Probability             Baserunning            
    Cent%  Oppo%             WPA  cWPA  RE24         RS%  SB% XBT%  
 0  45.1%  24.4%             0.9  0.2%   6.8         19%  50%  27%  
 1  47.9%  17.9%             1.3  1.1%  19.2         41%  77%  61%  
 2  51.8%  16.0%             3.0  2.0%  34.7         36%  84%  51%  
 3  49.6%  17.5%             5.3  3.3%  60.7         36%  80%  53%  
 4  51.8%  18.2%             NaN   NaN   NaN         30%  78%  42%  
 
 [5 rows x 26 columns]]

Baseball Reference is Tricky

This behavior is driven by the site identifying that we’re a bot, not a human – the HTML they serve to a human works great:

human_tables = pd.read_html('https://lukashager.netlify.app/econ-481/data/gunnar_henderson')
len(human_tables)

To get the table we want, we should specify the match keyword

pd.read_html(
    'https://lukashager.netlify.app/econ-481/data/gunnar_henderson',
    match = 'Oppo%'
)

[  Unnamed: 0_level_0 Unnamed: 1_level_0 Unnamed: 2_level_0 Unnamed: 3_level_0  \
                 Year                Age                 Tm                 Lg   
 0               2022                 21                BAL                 AL   
 1               2023                 22                BAL                 AL   
 2              2 Yrs              2 Yrs              2 Yrs              2 Yrs   
 3       MLB Averages       MLB Averages       MLB Averages       MLB Averages   
 
   Batting                     Batting Ratios         ... Batted Ball         \
      rOBA Rbat+  BAbip    ISO            HR%    SO%  ...       GB/FB  Pull%   
 0   0.348   128  0.333  0.181           3.0%  25.8%  ...        1.34  30.5%   
 1   0.349   125  0.306  0.234           4.5%  25.6%  ...        0.84  34.2%   
 2   0.349   125  0.311  0.225           4.2%  25.6%  ...        0.91  33.6%   
 3   0.320   100  0.293  0.159           3.0%  22.6%  ...        0.75  29.8%   
 
                 Win Probability             Baserunning            
    Cent%  Oppo%             WPA  cWPA  RE24         RS%  SB% XBT%  
 0  45.1%  24.4%             0.9  0.2%   6.8         19%  50%  27%  
 1  47.9%  17.9%             1.3  1.1%  19.2         41%  77%  61%  
 2  47.4%  19.0%             2.2  1.3%  26.0         37%  73%  56%  
 3  51.9%  18.3%             NaN   NaN   NaN         30%  78%  42%  
 
 [4 rows x 26 columns]]

Using APIs

Data Population on Page

It’s possible that a page is sending a request itself to get the data that it uses to populate a page
If possible, it’s more efficient to try to request that API directly instead of scraping the HTML
Can sometimes be hard – depending on the API, you may need to authenticate or pass cookies

Western States

The oldest and most prestigious ultramarathon in the United States
100 miles from Olympic Valley, CA to Auburn, CA
Used to be a horse race
- Gordy Ainsleigh ran it in 24:42 in 1974

Gordy Ainsleigh, courtesy of Western States 100

Western States Results

Stored in tabular format – can we use pd.read_html?

pd.read_html('https://ultrasignup.com/results_event.aspx?did=97204')

[         0       1       2       3
 0      NaN     NaN     NaN     NaN
 1   2024.0  2023.0  2022.0  2021.0
 2   2020.0  2019.0  2018.0  2017.0
 3   2016.0  2015.0  2014.0  2013.0
 4   2012.0  2011.0  2010.0  2009.0
 5   2007.0  2006.0  2005.0  2004.0
 6   2003.0  2002.0  2001.0  2000.0
 7   1999.0  1998.0  1997.0  1996.0
 8   1995.0  1994.0  1993.0  1992.0
 9   1991.0  1990.0  1989.0  1988.0
 10  1987.0  1986.0  1985.0  1984.0
 11  1983.0  1982.0  1981.0  1980.0
 12  1979.0  1978.0  1977.0  1976.0
 13  1974.0     NaN     NaN     NaN
 14     NaN     NaN     NaN     NaN]

No – they’re not even serving us the table.

Network Tab in Chrome

This shows what a webpage is doing as it loads
- For example, we can see the images it loads, the scripts it deploys
- Importantly: can see the APIs it requests
In this case, we see that it’s requesting a json that looks like what we want

Network Tab json Request

The JSON

Requesting Directly

If the page makes this request, it stands to reason that we can as well

url_2023 = 'https://ultrasignup.com/service/events.svc/results/87878/1/json?'
ws_req = requests.get(url_2023)
ws_req.ok

True

Passing `json` to `DataFrame`

print(pd.DataFrame(ws_req.json()))

     age  age_rank agegroup  bib               city distance_time  drilldown  \
0     26         0    20-29   34           Missoula             0          0   
1     31         0    30-39              Cedar City             0          0   
2     29         0    20-29   19          Massillon             0          0   
3     38         0    30-39                Portland             0          0   
4     30         0    30-39                 Boulder             0          0   
..   ...       ...      ...  ...                ...           ...        ...   
378   41         0    40-49  332  Piovene Rocchette             0          0   
379   45         0    40-49  216          Lane Cove             0          0   
380   27         0    20-29  314           Hamilton             0          0   
381   53         0    50-59  170             Tucson             0          0   
382   31         0    30-39  120         Atascadero             0          0   

    firstname formattime gender  ...  lastname participant_id photo_count  \
0        Adam   15:13:48      M  ...  Peterman        1854799           0   
1      Hayden   15:47:27      M  ...     Hawks        2310798           0   
2       Arlen   15:56:17      M  ...     Glick        2392278           0   
3       Tyler   15:57:10      M  ...     Green        2392344           0   
4        Drew   16:09:00      M  ...    Holmen        1075712           0   
..        ...        ...    ...  ...       ...            ...         ...   
378   Daniele          0      M  ...  Sperotto        1855275           0   
379      Dean          0      M  ...    Israel        1855276           0   
380      Kurt          0      M  ...     Rider        1855285           0   
381     Tyler          0      M  ...      Ford        1964840           0   
382    Jordan          0      M  ...   Collins        1988320           0   

     place  prior_count  race_count  runner_rank  state  status      time  
0        1            0           0        95.61     MT       1  54828000  
1        2            0           0        97.37     UT       1  56847000  
2        3            0           0        94.35     OH       1  57377000  
3        4            0           0        95.41     OR       1  57430000  
4        5            0           0        96.13     CO       1  58140000  
..     ...          ...         ...          ...    ...     ...       ...  
378      0            0           0        62.43    ITA       2         0  
379      0            0           0        63.44    AUS       2         0  
380      0            0           0        46.43    AUS       2         0  
381      0            0           0        56.44     AZ       2         0  
382      0            0           0        67.51     CA       2         0  

[383 rows x 22 columns]

Scraping Multiple Pages

Data Stored on Multiple Pages

It’s uncommon that we get everything we need from one page
- At that point, unclear what the value of scraping is
Often have to iterate over multiple pages and combine results

Scraping Multiple Years of WS Results

Say we wanted results from 2022 and 2023
Inspecting our URL leads us to believe that all we need to change is the event ID
We can grab the 2022 and 2023 IDs:

ev_ids = ['87878', '97204']

Plugging Into API

Warning

ALWAYS put breaks in your code. If you do not, you may crash the site and get into serious trouble.

import time

df_list = []
for ev_id in ev_ids:
    r = requests.get(
        f'https://ultrasignup.com/service/events.svc/results/{ev_id}/1/json?'
    )
    df = pd.DataFrame(r.json())
    df['event_id'] = ev_id
    df_list.append(df)
    time.sleep(5)

print(pd.concat(df_list).head(10))

Plugging Into API

   age  age_rank agegroup bib              city distance_time  drilldown  \
0   26         0    20-29  34          Missoula             0          0   
1   31         0    30-39            Cedar City             0          0   
2   29         0    20-29  19         Massillon             0          0   
3   38         0    30-39              Portland             0          0   
4   30         0    30-39               Boulder             0          0   
5   46         0    40-49  13         Prevessin             0          0   
6   38         0    30-39  15            Annecy             0          0   
7   37         0    30-39      Colorado Springs             0          0   
8   27         0    20-29               Challis             0          0   
9   41         0    40-49  29            Woburn             0          0   

  firstname formattime gender  ...  participant_id photo_count place  \
0      Adam   15:13:48      M  ...         1854799           0     1   
1    Hayden   15:47:27      M  ...         2310798           0     2   
2     Arlen   15:56:17      M  ...         2392278           0     3   
3     Tyler   15:57:10      M  ...         2392344           0     4   
4      Drew   16:09:00      M  ...         1075712           0     5   
5   Ludovic   16:20:02      M  ...         2392267           0     6   
6   Vincent   16:28:22      M  ...         2059474           0     7   
7      Alex   16:28:34      M  ...         2392547           0     8   
8      Cody   16:29:38      M  ...         2392322           0     9   
9     Scott   16:35:23      M  ...         2392300           0    10   

   prior_count  race_count  runner_rank  state  status      time event_id  
0            0           0        95.61     MT       1  54828000    87878  
1            0           0        97.37     UT       1  56847000    87878  
2            0           0        94.35     OH       1  57377000    87878  
3            0           0        95.41     OR       1  57430000    87878  
4            0           0        96.13     CO       1  58140000    87878  
5            0           0        94.18    FRA       1  58802000    87878  
6            0           0        91.67              1  59302000    87878  
7            0           0        95.29     CO       1  59314000    87878  
8            0           0        93.84     ID       1  59378000    87878  
9            0           0        93.86     MA       1  59723000    87878  

[10 rows x 23 columns]

Getting Event IDs

Kalvin made a great point – how would we get the identifiers programmatically?

r = requests.get('https://ultrasignup.com/results_event.aspx?did=97204')
bs = BeautifulSoup(r.text)
r.ok

True

Getting the Elements

table = bs.find('table', {'id':'ContentPlaceHolder1_dlYears'})
rows = table.find_all('tr')
elements = [row.find_all('td') for row in rows]
years = [y.text.strip() for x in elements[1:-2] for y in x]
links = [y.find('a')['href'] for x in elements[1:-2] for y in x]
ids = [re.findall('\d+', x)[0] for x in links]

Putting Together

print(pd.DataFrame({'year': years, 'link': links, 'ev_id': ids}))

    year                            link   ev_id
0   2024  /results_event.aspx?did=108752  108752
1   2023   /results_event.aspx?did=97204   97204
2   2022   /results_event.aspx?did=87878   87878
3   2021   /results_event.aspx?did=79446   79446
4   2020   /results_event.aspx?did=71208   71208
5   2019   /results_event.aspx?did=61359   61359
6   2018   /results_event.aspx?did=51243   51243
7   2017   /results_event.aspx?did=41765   41765
8   2016   /results_event.aspx?did=34773   34773
9   2015   /results_event.aspx?did=30033   30033
10  2014   /results_event.aspx?did=24962   24962
11  2013   /results_event.aspx?did=17746   17746
12  2012   /results_event.aspx?did=14050   14050
13  2011   /results_event.aspx?did=10804   10804
14  2010    /results_event.aspx?did=5752    5752
15  2009    /results_event.aspx?did=4742    4742
16  2007     /results_event.aspx?did=771     771
17  2006    /results_event.aspx?did=5705    5705
18  2005    /results_event.aspx?did=5706    5706
19  2004    /results_event.aspx?did=5707    5707
20  2003    /results_event.aspx?did=5708    5708
21  2002    /results_event.aspx?did=6734    6734
22  2001    /results_event.aspx?did=6733    6733
23  2000    /results_event.aspx?did=6732    6732
24  1999    /results_event.aspx?did=6731    6731
25  1998    /results_event.aspx?did=6730    6730
26  1997    /results_event.aspx?did=6729    6729
27  1996    /results_event.aspx?did=6728    6728
28  1995    /results_event.aspx?did=6727    6727
29  1994    /results_event.aspx?did=6726    6726
30  1993    /results_event.aspx?did=6725    6725
31  1992    /results_event.aspx?did=6724    6724
32  1991    /results_event.aspx?did=6723    6723
33  1990    /results_event.aspx?did=6722    6722
34  1989    /results_event.aspx?did=6721    6721
35  1988    /results_event.aspx?did=6720    6720
36  1987    /results_event.aspx?did=6719    6719
37  1986    /results_event.aspx?did=6718    6718
38  1985    /results_event.aspx?did=6717    6717
39  1984    /results_event.aspx?did=6716    6716
40  1983    /results_event.aspx?did=6715    6715
41  1982    /results_event.aspx?did=6714    6714
42  1981    /results_event.aspx?did=6713    6713
43  1980    /results_event.aspx?did=6712    6712
44  1979    /results_event.aspx?did=6711    6711
45  1978    /results_event.aspx?did=6710    6710
46  1977    /results_event.aspx?did=6709    6709
47  1976    /results_event.aspx?did=6708    6708

Authentication

Problem: Sites Recognize Bots

We’ve seen a few times that websites will recognize that we’re not people
This is due to a few possible reasons:
- Our requests don’t have payloads that indicate that we’re humans
- There’s some sort of human verification on the page
The former we can resolve – the latter, not so much

Examples

Baseball Reference hides their data in comments
UltraSignup hides their data entirely

Potential Solution: Authenticate

Some sites will return different HTML to scrapers depending on whether or not the scraper is an authenticated account
In this case, we can do the following:

Send a POST request to the site to authenticate
Use a GET request to get the data from the site/API

Issue

I don’t want to hardcode any of my passwords into this presentation
I don’t have a good example where I can create a throwaway account

Very Simple – Olympus

Thanks to Real Python there’s a practice login page located here
We can inspect the form and see what the POST request needs to look like to authenticate

Exercise: Olympus

Find the POST request that occurs on this page when you log in using username “zeus” and password “ThunderDude”. Remember to use Chrome and use CTRL+SHIFT+C.

Olympus Headers

Headers

Olympus Payload

Payload

Constructing the Request

This is all we need to get the HTML behind the login:

headers = {'user': 'zeus', 'pwd': 'ThunderDude'}
r = requests.post(
    'http://olympus.realpython.org/login',
    data = headers
)

Exercise: Extracting Links

Use BeautifulSoup to extract all the links from the logged-in Olympus page.

Solutions: Extracting Links

We can use BeautifulSoup’s find_all with the a tag.

olympus_bs = BeautifulSoup(r.text)
[x['href'] for x in olympus_bs.find_all('a')]

['/profiles/aphrodite', '/profiles/poseidon', '/profiles/dionysus']

UW Economics Database

Goal (Scraping Non-Tabular Data)

We want to list every graduate student in Economics at UW as well as whatever data the Department of Economics makes publicly available about them.

Exercise: Strategy

Come up with the strategy we should use to accomplish this task. Don’t write any code, but figure out what pages would be helpful, and think about what sort of code we’ll need to write.

Approach

List all of the links on this page
For each link, extract all of the relevant data from wherever it’s stored.

Step 1

r1 = requests.get('https://econ.washington.edu/people/graduate-student')
assert r1.ok
r1_bs = BeautifulSoup(r1.text)

To get all the links, we could try what we did before:

links = r1_bs.find_all('a')
links

[<a class="visually-hidden focusable skip-link" href="#main-content">
     Skip to main content
   </a>,
 <a class="uw-link" href="https://www.washington.edu">
 <div class="w-logo"><svg aria-labelledby="W_Title" data-name="W Logo" id="W_Logo" role="img" viewbox="0 0 120.29 80.88" xmlns="http://www.w3.org/2000/svg"><title id="W_Title">Washington</title><path d="M88.27,0V14.58H98.52L87.08,57.19,72.87,0H58.06L42.68,57.19,32.15,14.58H42.82V0H0V14.58H9.49s16.36,65.51,16.57,66.3H49L60,39.17c6.83,27.31,6.49,25.58,10.43,41.72h23c0.2-.78,17.43-66.3,17.43-66.3h9.41V0h-32Z" fill="#fff"></path></svg></div>
 <div class="university-wordmark show-for-medium-up"><svg aria-labelledby="UW_Title" data-name="University of Washington" id="UW" role="img" viewbox="0 0 207.59 13.98" xmlns="http://www.w3.org/2000/svg"><title id="UW_Title">University of Washington</title><path d="M202.28,10.91V3.2l5.35,8.6h0.53V2.4L209,1.51h-2.69l0.89,0.89V8.94l-4.64-7.43h-2.13l0.87,0.89v8.52l-0.87.89h2.69Zm-8,.33c-2.06,0-2.77-2.46-2.77-4.59s0.72-4.59,2.77-4.59,2.79,2.46,2.79,4.59-0.72,4.59-2.79,4.59M190.2,6.66c0,2.62,1.48,5.37,4.12,5.37s4.13-2.76,4.13-5.37S197,1.28,194.32,1.28,190.2,4,190.2,6.66M185.55,11V2.28h2l1.1,1.1V1.51h-7.24V3.38l1.12-1.1h1.94V11l-0.8.82h2.76Zm-7.62-3.86v3.31a2.22,2.22,0,0,1-1.69.73c-2.28,0-3.44-2.3-3.44-4.52s1.45-4.52,3.44-4.52a1.57,1.57,0,0,1,1.26.49l1.13,1.12V2a4.6,4.6,0,0,0-2.39-.59,5,5,0,0,0-4.78,5.29c0,2.72,2,5.29,4.78,5.29a4.49,4.49,0,0,0,2.83-.75V7.13l0.8-.82h-2.76Zm-15.26,3.79V3.2l5.42,8.61h0.46V2.4l0.89-.89h-2.69l0.89,0.89V8.94L163,1.51h-2.13l0.87,0.89v8.52l-0.87.89h2.69Zm-7.38-9.4,0.82,0.82V11l-0.82.82h2.77L157.23,11V2.33l0.82-.82h-2.77Zm-9.14,5.13h4.33V11l-0.82.82h2.78L151.6,11V2.33l0.82-.82h-2.78l0.82,0.82V5.87h-4.33V2.33L147,1.51H144.2L145,2.33V11l-0.8.82H147L146.14,11V6.64Zm-4.64,2.3a2.28,2.28,0,0,0-1-2l-2.09-1.52a2.43,2.43,0,0,1-1.14-1.76,1.45,1.45,0,0,1,1.48-1.46,5.94,5.94,0,0,1,.66.12l0.91,1,1-1.43-2-.38a1.63,1.63,0,0,0-.49-0.05A2.59,2.59,0,0,0,136.17,4a3,3,0,0,0,1.33,2.36l2.11,1.52a1.75,1.75,0,0,1,.73,1.5,1.7,1.7,0,0,1-1.71,1.82,1.73,1.73,0,0,1-.72-0.14l-1-1.15-0.89,1.61,2,0.38a3.5,3.5,0,0,0,.61.07,3,3,0,0,0,2.86-3M130,3.8l1.41,4.26h-2.88ZM127.5,11l0.75-2.18h3.4L132.37,11l-0.8.8h2.91L133.66,11l-3.21-9.46h-0.52L126.66,11l-0.84.84h2.48Zm-10.95.8,2.36-7.45,2.29,7.45h0.52l3-9.47,0.84-.82h-2.48l0.8,0.8-2.18,7.12-2.11-7.12,0.8-.8h-2.91l0.82,0.82L118.49,3l-2,6.42-2.11-7.12,0.8-.8h-2.91l0.84,0.82L116,11.8h0.52ZM101.3,5.63L100.9,6l0,0.14h1.2c-0.19,1.31-.36,2.44-0.64,4.26-0.39,2.71-.72,3.67-1,3.9a0.53,0.53,0,0,1-.36.13,2,2,0,0,1-.64-0.22,0.34,0.34,0,0,0-.41.1,0.88,0.88,0,0,0-.25.49,0.58,0.58,0,0,0,.62.43,2.44,2.44,0,0,0,1.59-1c0.49-.61,1.16-2,1.72-5.29,0.1-.61.22-1.22,0.46-2.81L104.74,6l0.32-.38h-1.71c0.43-2.68.8-3.49,1.42-3.49a1.57,1.57,0,0,1,1.12.48,0.29,0.29,0,0,0,.41,0,0.86,0.86,0,0,0,.29-0.54,0.79,0.79,0,0,0-.87-0.61,3.22,3.22,0,0,0-2.33,1.22,6.55,6.55,0,0,0-1.15,3H101.3Zm-6.52,3.8c0-2.15,1.09-3.31,1.54-3.49a1.35,1.35,0,0,1,.48-0.12c0.72,0,1.14.55,1.14,1.67,0,1.88-1,3.59-1.57,3.8a1.43,1.43,0,0,1-.45.1c-0.81,0-1.15-.87-1.15-2m2.48-4.15a3.45,3.45,0,0,0-1.57.52,4.54,4.54,0,0,0-2.09,4,1.89,1.89,0,0,0,1.83,2.15,3.73,3.73,0,0,0,2.16-1A5.2,5.2,0,0,0,99.11,7.3a1.78,1.78,0,0,0-1.86-2m-16.8-3L83.05,7v4l-0.82.82H85L84.19,11V7l2.65-4.68,0.86-.84H85.15l0.8,0.8L83.78,6.17,81.73,2.31l0.8-.8H79.61ZM75,11V2.28h2l1.1,1.1V1.51H70.84V3.38L72,2.28h1.94V11l-0.8.82h2.76ZM65.69,1.51l0.82,0.82V11l-0.82.82h2.78L67.64,11V2.33l0.82-.82H65.69Zm-3,7.43a2.28,2.28,0,0,0-1-2L59.65,5.45a2.42,2.42,0,0,1-1.13-1.76A1.45,1.45,0,0,1,60,2.23a5.89,5.89,0,0,1,.66.12l0.91,1,1-1.43-2-.38A1.61,1.61,0,0,0,60,1.46,2.59,2.59,0,0,0,57.38,4a3,3,0,0,0,1.33,2.36l2.11,1.52a1.75,1.75,0,0,1,.73,1.5,1.7,1.7,0,0,1-1.71,1.82A1.72,1.72,0,0,1,59.13,11l-1-1.15-0.89,1.61,2,0.38a3.5,3.5,0,0,0,.61.07,3,3,0,0,0,2.86-3M49.48,2.28h1.26a2,2,0,0,1,2,2.06,1.93,1.93,0,0,1-2,1.9H49.48v-4Zm0,8.71V7H51l2.49,4.8h1.71L54.43,11l-2.3-4.22a2.7,2.7,0,0,0,1.94-2.41,3.08,3.08,0,0,0-3.33-2.84H47.54l0.8,0.82V11l-0.8.82H50.3Zm-4.62.82V9.92L43.76,11H40.63V6.71H43l0.82,0.82V5.12L43,5.94H40.63V2.28h3.12l1.1,1.1V1.51H38.7l0.8,0.82V11l-0.8.82h6.16ZM31.51,1.51H28.59l0.84,0.82,2.9,9.47h0.52l3-9.47,0.84-.82H34.19L35,2.31,32.82,9.43,30.7,2.31Zm-8,0,0.82,0.82V11l-0.82.82h2.77L25.47,11V2.33l0.82-.82H23.51Zm-9.58,9.4V3.2l5.52,8.6h0.36V2.4l0.89-.89H18L18.91,2.4V8.94L14.27,1.51H12.14L13,2.4v8.52l-0.87.89h2.69ZM2.35,9a3.07,3.07,0,0,0,3.19,3A3.21,3.21,0,0,0,8.87,9V2.4l0.87-.89H7.06L7.95,2.4V9a2.11,2.11,0,0,1-2.13,2.13A2.21,2.21,0,0,1,3.55,9V2.4l0.89-.89h-3L2.35,2.4V9Z" fill="#fff" transform="translate(-1.46 -1.28)"></path></svg></div>
 </a>,
 <a class="artsci-link show-for-medium-up" href="https://artsci.washington.edu">College of Arts &amp; Sciences</a>,
 <a data-drupal-link-system-path="node/636" href="/support-us" title="">Make a Gift</a>,
 <a href="https://uw.edu/directory" title="">Directories</a>,
 <a href="https://uw.edu/maps" title="">Maps</a>,
 <a href="https://my.uw.edu" title="">MyUW</a>,
 <a class="simplesamlphp-auth-login-link" href="/saml_login" title="UWNetID Login">UWNetID Login</a>,
 <a aria-controls="search-field" aria-expanded="false" href="#" id="search-toggle">Search</a>,
 <a href="/" title="Department of Economics Home">Department of Economics</a>,
 <a href="#"><span>Menu</span></a>,
 <a data-drupal-link-system-path="people" href="/people">People</a>,
 <a data-drupal-link-system-path="people/faculty" href="/people/faculty" title="">Faculty</a>,
 <a data-drupal-link-system-path="node/732" href="/visiting-scholars">Visitors</a>,
 <a data-drupal-link-system-path="people/staff" href="/people/staff" title="">Staff</a>,
 <a aria-current="page" class="is-active" data-drupal-link-system-path="people/graduate-student" href="/people/graduate-student" title="">Graduate Students</a>,
 <a aria-current="page" class="is-active" data-drupal-link-system-path="people/graduate-student" href="/people/graduate-student" title="">All Graduate Students</a>,
 <a data-drupal-link-system-path="node/1057" href="/people/graduate-student/first-year-phd-student">First Year PhD Students</a>,
 <a data-drupal-link-system-path="node/1058" href="/people/graduate-student/job-market-candidate">Job Market Candidates</a>,
 <a data-drupal-link-system-path="node/644" href="/alumni">Alumni</a>,
 <a data-drupal-link-system-path="node/644" href="/alumni" title="">Alumni News</a>,
 <a data-drupal-link-system-path="node/683" href="/notable-economics-alumni">Notable Economics Alumni</a>,
 <a data-drupal-link-system-path="node/678" href="/economics-alumni-updates">Economics Alumni Updates</a>,
 <a data-drupal-link-system-path="webform/alumni_update" href="/alumni-update">Share Your Update</a>,
 <a data-drupal-link-system-path="node/1061" href="/people/alumni/visiting-committee-member">Visiting Committee Members</a>,
 <a data-drupal-link-system-path="node/634" href="/programs-courses">Programs &amp; Courses</a>,
 <a data-drupal-link-system-path="node/638" href="/undergraduate-programs">Undergraduate</a>,
 <a data-drupal-link-system-path="node/690" href="/economics-degrees">Degree Options</a>,
 <a data-drupal-link-system-path="node/695" href="/apply-economics-major">How to Apply</a>,
 <a data-drupal-link-system-path="node/696" href="/courses-and-registration" title="">Course Planning &amp; Registration</a>,
 <a data-drupal-link-system-path="node/726" href="/special-programs">Special Programs</a>,
 <a data-drupal-link-system-path="node/705" href="/resources-students">Resources for Students</a>,
 <a data-drupal-link-system-path="node/709" href="/information-graduating-seniors">Graduation</a>,
 <a data-drupal-link-system-path="node/689" href="/economics-undergraduate-board">Economics Undergraduate Board</a>,
 <a data-drupal-link-system-path="node/727" href="/internships-careers">Internships &amp; Careers</a>,
 <a data-drupal-link-system-path="node/639" href="/graduate-programs">Graduate</a>,
 <a data-drupal-link-system-path="node/653" href="/graduate-program-overview">Overview</a>,
 <a data-drupal-link-system-path="node/655" href="/graduate-application-procedure-forms">How to Apply</a>,
 <a data-drupal-link-system-path="node/654" href="/graduate-admission-requirements">Admission Requirements</a>,
 <a data-drupal-link-system-path="node/658" href="/academic-student-employees">Academic Student Employees</a>,
 <a data-drupal-link-system-path="node/656" href="/graduate-student-financial-assistance">Financial Assistance</a>,
 <a data-drupal-link-system-path="node/657" href="/job-placement">Job Placement</a>,
 <a data-drupal-link-system-path="node/472" href="/news/2016/11/14/selected-publications-recent-phd-graduates" title="">Recent Publications</a>,
 <a data-drupal-link-system-path="node/731" href="/admission-faqs">FAQs</a>,
 <a data-drupal-link-system-path="courses" href="/courses" title="">Courses</a>,
 <a data-drupal-link-system-path="courses" href="/courses" title="">Current &amp; Upcoming</a>,
 <a data-drupal-link-system-path="node/696" href="/courses-and-registration">Registering for Courses</a>,
 <a data-drupal-link-system-path="node/698" href="/add-code-requests-policy">Add Code Requests Policy</a>,
 <a data-drupal-link-system-path="node/697" href="/undergraduate-course-syllabi">Undergraduate Course Syllabi</a>,
 <a data-drupal-link-system-path="node/694" href="/policy-academic-conduct">Policy on Academic Conduct</a>,
 <a data-drupal-link-system-path="node/713" href="/study-abroad">Study Abroad</a>,
 <a data-drupal-link-system-path="node/713" href="/study-abroad" title="">General Information</a>,
 <a data-drupal-link-system-path="node/714" href="/study-abroad-program-course-equivalents">Course Equivalents</a>,
 <a data-drupal-link-system-path="node/669" href="/economics-outreach" title="">Outreach</a>,
 <a data-drupal-link-system-path="node/671" href="/undergraduate-mentorship-program">Undergraduate Mentorship Program</a>,
 <a data-drupal-link-system-path="node/675" href="/department-economics-mentors">Department of Economics Mentors</a>,
 <a data-drupal-link-system-path="node/672" href="/mentor-resource-library">Mentor Resource Library</a>,
 <a data-drupal-link-system-path="node/673" href="/mentors-faqs">Mentors' FAQs</a>,
 <a data-drupal-link-system-path="node/674" href="/mentorship-program-information-students">Mentorship Info for Students</a>,
 <a data-drupal-link-system-path="node/676" href="/resources-mentoring-international-students">Resources for Mentoring International Students</a>,
 <a data-drupal-link-system-path="node/681" href="/distinguished-alumni-award" title="">Awards</a>,
 <a data-drupal-link-system-path="node/681" href="/distinguished-alumni-award">Distinguished Alumnus</a>,
 <a data-drupal-link-system-path="node/777" href="/henry-t-buechel-awards">Henry T. Buechel</a>,
 <a data-drupal-link-system-path="node/686" href="/kathy-gehrt-memorial-economics-volunteer-service-award">Kathy Gehrt Memorial</a>,
 <a data-drupal-link-system-path="node/778" href="/undergraduate-scholarship-recipients">Undergraduate Scholarship Recipients</a>,
 <a data-drupal-link-system-path="node/779" href="/graduate-student-awards">Graduate Student Awards</a>,
 <a data-drupal-link-system-path="node/738" href="/diversity-equity-and-inclusion">Diversity, Equity and Inclusion</a>,
 <a data-drupal-link-system-path="node/748" href="/biennial-endowed-milliman-lecture-economics">Endowed Milliman Lecture</a>,
 <a data-drupal-link-system-path="node/669" href="/economics-outreach">Economics Outreach</a>,
 <a data-drupal-link-system-path="node/682" href="/public-lectures-and-lifelong-learning">Lifelong Learning</a>,
 <a data-drupal-link-system-path="node/670" href="/uw-women-economics">UW Women in Economics</a>,
 <a data-drupal-link-system-path="node/679" href="/economics-visiting-committee">Visiting Committee</a>,
 <a data-drupal-link-system-path="node/648" href="/research">Research</a>,
 <a data-drupal-link-system-path="node/753" href="/research-articles">Articles</a>,
 <a data-drupal-link-system-path="node/752" href="/research-bookschapters">Books/Chapters</a>,
 <a data-drupal-link-system-path="node/754" href="/research-dissertations">Dissertations</a>,
 <a data-drupal-link-system-path="news" href="/news" title="">News &amp; Events</a>,
 <a data-drupal-link-system-path="news/recent" href="/news/recent" title="">Recent News</a>,
 <a data-drupal-link-system-path="node/643" href="/newsletters">Newsletter</a>,
 <a data-drupal-link-system-path="calendar" href="/calendar" title="">Events Calendar</a>,
 <a data-drupal-link-system-path="node/724" href="/brownbag-seminars">Brownbag Seminars</a>,
 <a data-drupal-link-system-path="node/721" href="/econometrics-seminar-series">Econometrics Seminars</a>,
 <a data-drupal-link-system-path="node/740" href="/economics-major-info-session">Economics Major Info Sessions</a>,
 <a data-drupal-link-system-path="node/720" href="/international-economics-and-macroeconomics-seminar-series">International and Macroeconomics Seminars</a>,
 <a data-drupal-link-system-path="node/722" href="/joint-seminar-development-economics-series">Joint Seminar in Development Economics</a>,
 <a data-drupal-link-system-path="node/723" href="/microeconomics-seminar-series">Microeconomics Seminars</a>,
 <a data-drupal-link-system-path="node/748" href="/biennial-endowed-milliman-lecture-economics" title="">Milliman Endowed Lectures</a>,
 <a data-drupal-link-system-path="node/687" href="/seminar-series">Seminar Series</a>,
 <a data-drupal-link-system-path="node/635" href="/resources">Resources</a>,
 <a data-drupal-link-system-path="node/705" href="/resources-students">Advising</a>,
 <a data-drupal-link-system-path="node/727" href="/internships-careers" title="">Internships &amp; Careers</a>,
 <a data-drupal-link-system-path="node/657" href="/job-placement" title="">Grad Program Job Placement</a>,
 <a data-drupal-link-system-path="node/706" href="/internships">Undergrad Internships</a>,
 <a data-drupal-link-system-path="node/641" href="/career-planning">Undergrad Career Planning</a>,
 <a data-drupal-link-system-path="node/718" href="/research-and-computing-resources">Research and Computing</a>,
 <a data-drupal-link-system-path="node/719" href="/terminal-server-information">Terminal Server Information</a>,
 <a data-drupal-link-system-path="node/640" href="/scholarships-and-financial-support">Scholarships &amp; Financial Support</a>,
 <a href="https://uwnetid.sharepoint.com/:f:/r/sites/Econshare/Shared%20Documents/Faculty%20Intranet?csf=1&amp;web=1&amp;e=KQfbpo">Intranet</a>,
 <a data-drupal-link-system-path="node/633" href="/about">About</a>,
 <a data-drupal-link-system-path="node/633" href="/about" title="">Overview</a>,
 <a data-drupal-link-system-path="node/646" href="/history">History</a>,
 <a data-drupal-link-system-path="node/649" href="/health-and-safety-information">Health &amp; Safety</a>,
 <a data-drupal-link-system-path="node/637" href="/stay-connected">Stay Connected</a>,
 <a data-drupal-link-system-path="node/636" href="/support-us">Support Us</a>,
 <a data-drupal-link-system-path="node/651" href="/giving-faqs">Giving FAQs</a>,
 <a data-drupal-link-system-path="node/650" href="/support-our-faculty">Support Our Faculty</a>,
 <a data-drupal-link-system-path="node/652" href="/support-our-students">Support Our Students</a>,
 <a data-drupal-link-system-path="node/647" href="/contact">Contact Us</a>,
 <a data-drupal-link-system-path="node/636" href="/support-us" title="">Make a Gift</a>,
 <a href="https://uw.edu/directory" title="">Directories</a>,
 <a href="https://uw.edu/maps" title="">Maps</a>,
 <a href="https://my.uw.edu" title="">MyUW</a>,
 <a id="main-content"></a>,
 <a href="/">Home</a>,
 <a href="/people">People - Full Directory</a>,
 <a href="https://econ.washington.edu/people/amre-abken"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Amre Abken" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/amre_abken.jpg?itok=nAAZSEN7" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/amre-abken" hreflang="und">Amre Abken</a>,
 <a href="mailto:abken@uw.edu">abken@uw.edu</a>,
 <a href="https://econ.washington.edu/people/afsana-adiba"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Afsana" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/afsana_adiba.jpg?itok=-7gyCnAs" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/afsana-adiba" hreflang="und">Afsana Adiba</a>,
 <a href="https://econ.washington.edu/people/shabab-ahmed"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Profile Picture " height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/61274211_2415400908491744_2968890164765523968_n.jpeg?itok=aw6FfI3O" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/shabab-ahmed" hreflang="und">Shabab Ahmed</a>,
 <a href="mailto:sahmed95@uw.edu">sahmed95@uw.edu</a>,
 <a href="/fields/data-science">Data Science</a>,
 <a href="/fields/economics">Economics</a>,
 <a href="/fields/game-theory">Game Theory</a>,
 <a href="/fields/industrial-organization">Industrial Organization</a>,
 <a href="/fields/machine-learning">Machine Learning</a>,
 <a href="/fields/microeconomic-theory">Microeconomic Theory</a>,
 <a href="/fields/microeconomics">Microeconomics</a>,
 <a href="https://econ.washington.edu/people/ishmam-al-quddus"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="photo " height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/profile-images/Ishmam%20Al%20Quddus.jpg?h=fe095282&amp;itok=Ez5q4Mva" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/ishmam-al-quddus" hreflang="en">Ishmam Al Quddus</a>,
 <a href="https://econ.washington.edu/people/alireza-aminkhaki"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Alireza" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/alireza_aminkhaki.jpg?itok=wdHxMIt-" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/alireza-aminkhaki" hreflang="und">Alireza Aminkhaki</a>,
 <a href="mailto:aamink@uw.edu">aamink@uw.edu</a>,
 <a href="https://econ.washington.edu/people/erik-andersen"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Photo of Erik in front of cherry blossoms. " height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/profile-images/IMG_0335_0.jpeg?itok=NJRAGAuM" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/erik-andersen" hreflang="und">Erik Andersen</a>,
 <a href="mailto:eander46@uw.edu">eander46@uw.edu</a>,
 <a href="tel:%28415%29497-9136">(415) 497-9136</a>,
 <a href="/fields/contract-theory">Contract Theory</a>,
 <a href="/fields/development-economics">Development Economics</a>,
 <a href="/fields/environmental-economics">Environmental Economics</a>,
 <a href="/fields/game-theory">Game Theory</a>,
 <a href="/fields/software">Software</a>,
 <a href="https://econ.washington.edu/people/raied-arman"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Raied" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/raied_arman.jpg?itok=_VEA7Jf_" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/raied-arman" hreflang="und">Raied Arman</a>,
 <a href="https://econ.washington.edu/people/camille-bergeron"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="photo" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/profile-images/Camille%20Bergeron.jpg?h=df3de174&amp;itok=ZZh_gbTe" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/camille-bergeron" hreflang="en">Camille Bergeron</a>,
 <a href="https://econ.washington.edu/people/england-can"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="england photo" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/england.jpg?itok=mfWEA009" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/england-can" hreflang="und">England Can</a>,
 <a href="/fields/contract-theory">Contract Theory</a>,
 <a href="/fields/development-economics">Development Economics</a>,
 <a href="/fields/econometrics">Econometrics</a>,
 <a href="/fields/economic-history">Economic History</a>,
 <a href="/fields/game-theory">Game Theory</a>,
 <a href="/fields/industrial-organization">Industrial Organization</a>,
 <a href="/fields/microeconomic-theory">Microeconomic Theory</a>,
 <a href="https://econ.washington.edu/people/felipe-carneiro"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Felipe Carneiro" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/felipe_carneiro.jpg?itok=OlkOvhUb" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/felipe-carneiro" hreflang="und">Felipe Carneiro</a>,
 <a href="mailto:fcdf@uw.edu">fcdf@uw.edu</a>,
 <a href="/fields/applied-econometrics">Applied Econometrics</a>,
 <a href="/fields/applied-microeconomics">Applied Microeconomics</a>,
 <a href="/fields/specific/causal-inference">Causal Inference</a>,
 <a href="/fields/political-economy">Political Economy</a>,
 <a href="https://econ.washington.edu/people/zewei-chai"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Zewei Chai" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/zeweichai.jpg?h=225f10be&amp;itok=7g_njMSF" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/zewei-chai" hreflang="und">Zewei Chai</a>,
 <a href="https://econ.washington.edu/people/tynan-challenor"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Tynan" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/tynan_c.jpg?itok=iFpnq9Zw" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/tynan-challenor" hreflang="und">Tynan Challenor</a>,
 <a href="https://econ.washington.edu/people/hee-gee-chang"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Picture of Heejee Chang" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/43b74987-fcee-4e7d-bec7-4ef330d5d5b8.jpeg?h=0b55ea07&amp;itok=aOMfxwcX" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/hee-gee-chang" hreflang="und">Hee Gee Chang</a>,
 <a href="mailto:heejee@uw.edu">heejee@uw.edu</a>,
 <a href="https://econ.washington.edu/people/cheng-chen"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Cheng_Profile_Picture" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/profile_pic.jpeg?h=3e13c41b&amp;itok=7GyZ3Ktj" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/cheng-chen" hreflang="und">Cheng Chen</a>,
 <a href="mailto:cchen09@uw.edu">cchen09@uw.edu</a>,
 <a href="/fields/industrial-organization">Industrial Organization</a>,
 <a href="/fields/microeconomics">Microeconomics</a>,
 <a href="/fields/urban-studies">Urban Studies</a>,
 <a href="https://econ.washington.edu/people/weiyue-chen"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="File Photo" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/1085f89c-9cb0-43eb-a590-02948c13dc1f.jpeg?itok=55oPtFv4" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/weiyue-chen" hreflang="und">Weiyue Chen</a>,
 <a href="mailto:wchen97@uw.edu">wchen97@uw.edu</a>,
 <a href="/fields/econometrics">Econometrics</a>,
 <a href="/fields/macroeconomics">Macroeconomics</a>,
 <a href="/fields/time-series">Time Series</a>,
 <a href="/people/zihao-chen" hreflang="und">Zihao Chen</a>,
 <a href="mailto:zchen05@uw.edu">zchen05@uw.edu</a>,
 <a href="/fields/financial-economics">Financial Economics</a>,
 <a href="/fields/international-finance">International Finance</a>,
 <a href="/fields/macroeconomics">Macroeconomics</a>,
 <a href="/fields/time-series">Time Series</a>,
 <a href="https://econ.washington.edu/people/changhun-choi"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Changhun Choi" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/changhun_choi.jpg?itok=jhYwFb0Q" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/changhun-choi" hreflang="und">Changhun Choi</a>,
 <a href="mailto:cchoi83@uw.edu">cchoi83@uw.edu</a>,
 <a href="https://econ.washington.edu/people/kisan-choi"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="man in front of trees" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/profile-images/IMG_6126%205%20-%20Kisan%20Choi_0.JPG?itok=YfGuAwVg" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/kisan-choi" hreflang="und">Kisan Choi</a>,
 <a href="mailto:kisanc@uw.edu">kisanc@uw.edu</a>,
 <a href="/fields/labor-economics">Labor Economics</a>,
 <a href="/fields/macroeconomics">Macroeconomics</a>,
 <a href="/fields/specific/wealth-inequality">Wealth Inequality</a>,
 <a href="/people/yoon-choi" hreflang="und">Yoon Choi</a>,
 <a href="mailto:lemineml@uw.edu">lemineml@uw.edu</a>,
 <a href="https://econ.washington.edu/people/hae-yun-choung"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Hae Yun Choung" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/hae_yun_choung.jpg?itok=X6tP6t8p" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/hae-yun-choung" hreflang="und">Hae Yun Choung</a>,
 <a href="mailto:hychoung@uw.edu">hychoung@uw.edu</a>,
 <a href="https://econ.washington.edu/people/austin-craig"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Austin Craig University of Washington" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/linkedin_photo.jpg?itok=UxVjL48k" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/austin-craig" hreflang="und">Austin Craig</a>,
 <a href="mailto:craiga1@uw.edu">craiga1@uw.edu</a>,
 <a href="https://econ.washington.edu/people/ryan-cummings"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Ryan Cummings" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/ryan_cummings.jpg?itok=5RwR8SYL" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/ryan-cummings" hreflang="und">Ryan Cummings</a>,
 <a href="mailto:ryanmgc@uw.edu">ryanmgc@uw.edu</a>,
 <a href="https://econ.washington.edu/people/tianqi-terrence-dai"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="profile" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/profile-images/IMG_1235_0.jpeg?h=f81ebebc&amp;itok=-AWq_zsd" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/tianqi-terrence-dai" hreflang="und">Tianqi (Terrence) Dai</a>,
 <a href="mailto:tdai7@uw.edu">tdai7@uw.edu</a>,
 <a href="/fields/applied-microeconomics">Applied Microeconomics</a>,
 <a href="/fields/specific/causal-machine-learning">Causal Machine Learning</a>,
 <a href="/fields/development-economics">Development Economics</a>,
 <a href="/fields/machine-learning">Machine Learning</a>,
 <a href="https://econ.washington.edu/people/ram-utkarsh-das"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Ram" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/ram.jpg?itok=opXFc69n" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/ram-utkarsh-das" hreflang="und">Ram Utkarsh Das</a>,
 <a href="mailto:ram1999@uw.edu">ram1999@uw.edu</a>,
 <a href="https://econ.washington.edu/people/rajarshi-datta"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Econ-Website2" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/website-rd-sitting_cropped_0.jpeg?itok=lZC30Bn_" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/rajarshi-datta" hreflang="und">Rajarshi Datta</a>,
 <a href="mailto:rdatta2@uw.edu">rdatta2@uw.edu</a>,
 <a href="/fields/applied-econometrics">Applied Econometrics</a>,
 <a href="/fields/specific/heterogeneous-agent-dsges">Heterogeneous Agent DSGEs</a>,
 <a href="/fields/housing">Housing</a>,
 <a href="/fields/macroeconomics">Macroeconomics</a>,
 <a href="/fields/specific/time-series-forecasting">Time Series Forecasting</a>,
 <a href="/fields/specific/wealth-inequality">Wealth Inequality</a>,
 <a href="https://econ.washington.edu/people/aurelia-aochun-di"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Aurelia's portrait" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/aochun_fituw_0.png?itok=bERa7CWD" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/aurelia-aochun-di" hreflang="und">Aurelia (Aochun) Di</a>,
 <a href="mailto:adi317@uw.edu">adi317@uw.edu</a>,
 <a href="/fields/specific/behavioral-economics">Behavioral Economics</a>,
 <a href="/fields/development-economics">Development Economics</a>,
 <a href="/fields/education">Education</a>,
 <a href="/fields/labor-economics">Labor Economics</a>,
 <a href="https://econ.washington.edu/people/ananya-diwakant"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Ananya Diwakant" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/ananya_diwakant.jpg?itok=E8ilu3gl" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/ananya-diwakant" hreflang="und">Ananya Diwakant</a>,
 <a href="mailto:ananyadi@uw.edu">ananyadi@uw.edu</a>,
 <a href="/fields/applied-microeconomics">Applied Microeconomics</a>,
 <a href="/fields/development-economics">Development Economics</a>,
 <a href="/fields/econometrics">Econometrics</a>,
 <a href="https://econ.washington.edu/people/alfredo-effendy"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Profile Picture" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/profile_picture.jpg?itok=L699f-WJ" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/alfredo-effendy" hreflang="und">Alfredo Effendy</a>,
 <a href="mailto:aeffendy@uw.edu">aeffendy@uw.edu</a>,
 <a href="/fields/applied-econometrics">Applied Econometrics</a>,
 <a href="/fields/data-science">Data Science</a>,
 <a href="/fields/exchange-rates">Exchange Rates</a>,
 <a href="/fields/financial-economics">Financial Economics</a>,
 <a href="/fields/international-finance">International Finance</a>,
 <a href="/fields/machine-learning">Machine Learning</a>,
 <a href="/fields/macroeconomics">Macroeconomics</a>,
 <a href="https://econ.washington.edu/people/francis-haoyu-fang"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Francis (Haoyu) Fang" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/profile-images/WechatIMG58880%20%281%29.jpeg?itok=Cn7r_qot" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/francis-haoyu-fang" hreflang="en">Francis (Haoyu) Fang</a>,
 <a href="mailto:fhfang@uw.edu">fhfang@uw.edu</a>,
 <a href="/fields/specific/behavioral-economics">Behavioral Economics</a>,
 <a href="/fields/development-economics">Development Economics</a>,
 <a href="/fields/specific/household-finance">Household Finance</a>,
 <a href="https://econ.washington.edu/people/suvekshya-gautam"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Suvekshya Gautam" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/suvekshya_gautam.jpg?itok=kZbWr01N" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/suvekshya-gautam" hreflang="und">Suvekshya Gautam</a>,
 <a href="mailto:sgautam9@uw.edu">sgautam9@uw.edu</a>,
 <a href="https://econ.washington.edu/people/matthew-grimm"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="picture of me!" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/profile-images/_C5A0131.jpg?h=7ac1ee21&amp;itok=rtcA627h" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/matthew-grimm" hreflang="und">Matthew Grimm</a>,
 <a href="mailto:mgrimm5@uw.edu">mgrimm5@uw.edu</a>,
 <a href="/fields/contract-theory">Contract Theory</a>,
 <a href="/fields/financial-economics">Financial Economics</a>,
 <a href="/fields/game-theory">Game Theory</a>,
 <a href="/fields/microeconomic-theory">Microeconomic Theory</a>,
 <a href="https://econ.washington.edu/people/lukas-hager"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Lukas Hager" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/profile-images/headshot.jpg?itok=tVH6j5dZ" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/lukas-hager" hreflang="und">Lukas Hager</a>,
 <a href="mailto:lghhager@uw.edu">lghhager@uw.edu</a>,
 <a href="/fields/industrial-organization">Industrial Organization</a>,
 <a href="/fields/specific/interactive-learning">Interactive Learning</a>,
 <a href="/fields/machine-learning">Machine Learning</a>,
 <a href="/fields/online-learning">Online Learning</a>,
 <a href="https://econ.washington.edu/people/lucy-hong"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Headshot" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/profile-images/IMG_6953.jpeg?h=0189ec99&amp;itok=Ia_ONxq5" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/lucy-hong" hreflang="und">Lucy Hong</a>,
 <a href="mailto:luhong@uw.edu">luhong@uw.edu</a>,
 <a href="/fields/applied-microeconomics">Applied Microeconomics</a>,
 <a href="/fields/industrial-organization">Industrial Organization</a>,
 <a href="/fields/specific/regional-economics">Regional Economics</a>,
 <a href="https://econ.washington.edu/people/canyu-huang"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Canyu Huang" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/img_0534.jpg?h=718dbb15&amp;itok=zLjCdY_h" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/canyu-huang" hreflang="und">Canyu Huang</a>,
 <a href="mailto:cindycan@uw.edu">cindycan@uw.edu</a>,
 <a href="https://econ.washington.edu/people/yizhan-huang"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Yizhan Huang" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/yizhan_huang.jpg?h=d5617486&amp;itok=8fzucDMO" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/yizhan-huang" hreflang="und">Yizhan Huang</a>,
 <a href="mailto:yhuang10@uw.edu">yhuang10@uw.edu</a>,
 <a href="/fields/specific/international-macroeconomics-and-finance">International Macroeconomics and Finance</a>,
 <a href="/fields/specific/macroeconomics">Macroeconomics</a>,
 <a href="/fields/specific/monetary-economics">Monetary Economics</a>,
 <a href="https://econ.washington.edu/people/saya-ikegawa"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Saya Ikegawa" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/sayaikegawa.jpg?itok=RpSsH93R" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/saya-ikegawa" hreflang="und">Saya Ikegawa</a>,
 <a href="https://econ.washington.edu/people/ken-inosaki"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="smiling man in suit" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/profile-images/headshot%20-%20Ken%20Inosaki.JPG?itok=hMdRH1Np" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/ken-inosaki" hreflang="und">Ken Inosaki</a>,
 <a href="mailto:inosaki@uw.edu">inosaki@uw.edu</a>,
 <a href="/fields/applied-microeconomics">Applied Microeconomics</a>,
 <a href="/fields/data-science">Data Science</a>,
 <a href="/fields/econometrics">Econometrics</a>,
 <a href="/fields/industrial-organization">Industrial Organization</a>,
 <a href="https://econ.washington.edu/people/sudiksha-joshi"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Photo" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/screenshot_2023-01-28_at_1.58.27_pm_1.png?itok=xkatZivw" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/sudiksha-joshi" hreflang="und">Sudiksha Joshi</a>,
 <a href="mailto:joshi27s@uw.edu">joshi27s@uw.edu</a>,
 <a href="/fields/specific/labor-markets">Labor markets</a>,
 <a href="/fields/specific/monetary-policy">Monetary Policy</a>,
 <a href="/fields/specific/time-series-forecasting">Time Series Forecasting</a>,
 <a href="https://econ.washington.edu/people/jong-min-jung"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Jong photo" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/jong_min.jpg?h=0033339f&amp;itok=rL5d4QQJ" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/jong-min-jung" hreflang="und">Jong Min Jung</a>,
 <a href="mailto:jmjung@uw.edu">jmjung@uw.edu</a>,
 <a href="/fields/applied-microeconomics">Applied Microeconomics</a>,
 <a href="/fields/development-economics">Development Economics</a>,
 <a href="/fields/specific/health-economics">Health Economics</a>,
 <a href="https://econ.washington.edu/people/mahtab-karimi"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="woman smiling" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/profile-images/mahtab.png?itok=Jy_bY6cw" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/mahtab-karimi" hreflang="und">Mahtab Karimi</a>,
 <a href="mailto:mahtak@uw.edu">mahtak@uw.edu</a>,
 <a href="/fields/specific/cybersecurity">Cybersecurity</a>,
 <a href="/fields/specific/empirical-corporate-finance">Empirical corporate finance</a>,
 <a href="/fields/specific/natural-language-processing">Natural language processing</a>,
 <a href="/fields/specific/stock-based-compensation">Stock-based compensation</a>,
 <a href="https://econ.washington.edu/people/zahra-khanalizadeh"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Zahra profile picture" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/zahra_linkedin_2_0.jpg?itok=k0hUrBHK" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/zahra-khanalizadeh" hreflang="und">Zahra Khanalizadeh</a>,
 <a href="mailto:zkhnl@uw.edu">zkhnl@uw.edu</a>,
 <a href="https://econ.washington.edu/people/mikita-khurana"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Smiling woman in front of flowering cherry tree" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/profile-images/Khurana_photo%20-%20Mikita%20Khurana.jpg?itok=1gP1RjO2" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/mikita-khurana" hreflang="und">Mikita Khurana</a>,
 <a href="mailto:mikita13@uw.edu">mikita13@uw.edu</a>,
 <a href="/fields/specific/banking-and-financial-markets">Banking and Financial Markets</a>,
 <a href="/fields/international-finance">International Finance</a>,
 <a href="/fields/specific/international-macroeconomics">International Macroeconomics</a>,
 <a href="/fields/macroeconomics">Macroeconomics</a>,
 <a href="/fields/time-series">Time Series</a>,
 <a href="/people/hyun-ji-kim" hreflang="und">Hyun Ji Kim</a>,
 <a href="mailto:hjkim26@uw.edu">hjkim26@uw.edu</a>,
 <a href="https://econ.washington.edu/people/soobin-kim"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Soobin Kim" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/soobin_kim.jpg?itok=DLhWlSII" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/soobin-kim" hreflang="und">Soobin Kim</a>,
 <a href="https://econ.washington.edu/people/jiyoung-lee"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="img_0511.jpg" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/img_0511.jpg?h=953da0ba&amp;itok=Y4iS7hUf" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/jiyoung-lee" hreflang="und">Jiyoung Lee</a>,
 <a href="mailto:jylee269@uw.edu">jylee269@uw.edu</a>,
 <a href="/fields/international-trade">International Trade</a>,
 <a href="/fields/macroeconomics">Macroeconomics</a>,
 <a href="/people/yebeen-lee" hreflang="und">Yebeen Lee</a>,
 <a href="mailto:yebeen@uw.edu">yebeen@uw.edu</a>,
 <a href="https://econ.washington.edu/people/yuhan-lee"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Yuhan Lee" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/yuhan_lee.jpg?itok=2Q4V4XL3" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/yuhan-lee" hreflang="und">Yuhan Lee</a>,
 <a href="mailto:re90234@uw.edu">re90234@uw.edu</a>,
 <a href="/people/yufei-liao" hreflang="und">Yufei Liao</a>,
 <a href="mailto:yliao31@uw.edu">yliao31@uw.edu</a>,
 <a href="https://econ.washington.edu/people/wenqiu-ma"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Wenqiu Ma" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/wenqiu_ma.jpg?itok=P1uSGoyR" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/wenqiu-ma" hreflang="und">Wenqiu Ma</a>,
 <a href="mailto:wenqiuma@uw.edu">wenqiuma@uw.edu</a>,
 <a href="https://econ.washington.edu/people/resem-makan"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Photo portrait" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/img_7492_3.jpg?itok=ll8qRphG" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/resem-makan" hreflang="und">Resem Makan</a>,
 <a href="mailto:resmakan@uw.edu">resmakan@uw.edu</a>,
 <a href="/fields/culture">Culture</a>,
 <a href="/fields/development-economics">Development Economics</a>,
 <a href="/fields/specific/historical-economics">Historical Economics</a>,
 <a href="/fields/specific/institutions">Institutions</a>,
 <a href="/fields/political-economy">Political Economy</a>,
 <a href="https://econ.washington.edu/people/manya-malik"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Manya Malik" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/manya_malik.jpg?h=5366de83&amp;itok=nHtyKsMP" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/manya-malik" hreflang="und">Manya Malik</a>,
 <a href="mailto:mmalik16@uw.edu">mmalik16@uw.edu</a>,
 <a href="https://econ.washington.edu/people/daniel-moore"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Daniel Moore" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/profile-images/cropped.jpg?itok=vF-gxXjW" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/daniel-moore" hreflang="und">Daniel Moore</a>,
 <a href="mailto:dfmoore@uw.edu">dfmoore@uw.edu</a>,
 <a href="https://econ.washington.edu/people/yvonne-ng"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="smiling woman " height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/profile-images/Headshot_final%20-%20Yvonne%20Ng.jpg?itok=ZRO4IV5d" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/yvonne-ng" hreflang="und">Yvonne Ng</a>,
 <a href="mailto:ngyvonne@uw.edu">ngyvonne@uw.edu</a>,
 <a href="/fields/applied-microeconomics">Applied Microeconomics</a>,
 <a href="/fields/specific/empirical-industrial-organization">Empirical Industrial Organization</a>,
 <a href="/fields/financial-economics">Financial Economics</a>,
 <a href="/fields/specific/household-finance">Household Finance</a>,
 <a href="/fields/housing">Housing</a>,
 <a href="/fields/industrial-organization">Industrial Organization</a>,
 <a href="/fields/specific/public-finance">Public Finance</a>,
 <a href="/fields/specific/urban-and-real-estate-economics">Urban and Real Estate Economics</a>,
 <a href="/fields/urban-studies">Urban Studies</a>,
 <a href="https://econ.washington.edu/people/t-v-ninan"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Headshot of T V Ninan" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/profile-images/_MG_3096.jpg?h=73854f71&amp;itok=yVgOfez2" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/t-v-ninan" hreflang="und">T V Ninan</a>,
 <a href="mailto:tvninan@uw.edu">tvninan@uw.edu</a>,
 <a href="/fields/applied-microeconomics">Applied Microeconomics</a>,
 <a href="/fields/development-economics">Development Economics</a>,
 <a href="/fields/education">Education</a>,
 <a href="/fields/environmental-economics">Environmental Economics</a>,
 <a href="/fields/health">Health</a>,
 <a href="https://econ.washington.edu/people/rafia-nisat"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Rafia Nisat" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/rafianisat.jpg?itok=dLvQCzeK" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/rafia-nisat" hreflang="und">Rafia Nisat</a>,
 <a href="mailto:rnisat@uw.edu">rnisat@uw.edu</a>,
 <a href="https://econ.washington.edu/people/yigit-okar"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Yigit Okar" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/yigit_okar.jpg?itok=IH1TJcN9" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/yigit-okar" hreflang="und">Yigit Okar</a>,
 <a href="mailto:yokar@uw.edu">yokar@uw.edu</a>,
 <a href="/people/sangwoo-park" hreflang="und">Sangwoo Park</a>,
 <a href="https://econ.washington.edu/people/somin-park"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="park photo" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/somin_park_final.jpg?itok=rwy8P4y0" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/somin-park" hreflang="und">Somin Park</a>,
 <a href="/people/khashayar-pourtaheri" hreflang="und">Khashayar Pourtaheri</a>,
 <a href="https://econ.washington.edu/people/yuan-alice-qi"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Alice Qi" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/alice_qi.jpg?h=268b4947&amp;itok=MY1KaSqh" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/yuan-alice-qi" hreflang="und">Yuan (Alice) Qi</a>,
 <a href="mailto:ayqi@uw.edu">ayqi@uw.edu</a>,
 <a href="https://econ.washington.edu/people/tyson-ramirez"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Tyson photo" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/tyson.jpg?itok=9ow-2pm7" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/tyson-ramirez" hreflang="und">Tyson Ramirez</a>,
 <a href="mailto:tramir33@uw.edu">tramir33@uw.edu</a>,
 <a href="/fields/applied-econometrics">Applied Econometrics</a>,
 <a href="/fields/applied-microeconomics">Applied Microeconomics</a>,
 <a href="/fields/econometrics">Econometrics</a>,
 <a href="/fields/financial-economics">Financial Economics</a>,
 <a href="/fields/international-trade">International Trade</a>,
 <a href="/fields/specific/quantitative-finance">Quantitative Finance</a>,
 <a href="/people/kobe-rankich" hreflang="en">Kobe Rankich</a>,
 <a href="mailto:koberank@uw.edu">koberank@uw.edu</a>,
 <a href="https://econ.washington.edu/people/anirudh-ravishankar"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="photo" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/profile-images/Anirudh%20Ravishankar.jpg?h=592cf992&amp;itok=7JinUZQ4" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/anirudh-ravishankar" hreflang="en">Anirudh Ravishankar</a>,
 <a href="https://econ.washington.edu/people/monika-rohilla"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Monika" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/monika_rohilla.jpg?itok=070LJSoB" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/monika-rohilla" hreflang="und">Monika Rohilla</a>,
 <a href="https://econ.washington.edu/people/abby-schamp"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Abby Schamp" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/abby_schamp.jpg?itok=AnTBfqcg" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/abby-schamp" hreflang="und">Abby Schamp</a>,
 <a href="mailto:schampab@uw.edu">schampab@uw.edu</a>,
 <a href="https://econ.washington.edu/people/minyan-shen"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Minyan Shen" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/minyan_shen.jpg?itok=sd_fzMFq" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/minyan-shen" hreflang="und">Minyan Shen</a>,
 <a href="mailto:minyans@uw.edu">minyans@uw.edu</a>,
 <a href="https://econ.washington.edu/people/tsering-sherpa"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Photo" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/profile-images/IMG_5960.jpg?h=6b3ccdff&amp;itok=K9G--eHz" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/tsering-sherpa" hreflang="und">Tsering Sherpa</a>,
 <a href="mailto:tsherp@uw.edu">tsherp@uw.edu</a>,
 <a href="https://econ.washington.edu/people/ruiyuan-shi"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Shi" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/ruiyuan_shi_2275468.jpg?itok=BPw3lqH0" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/ruiyuan-shi" hreflang="und">Ruiyuan Shi</a>,
 <a href="mailto:rshi32@uw.edu">rshi32@uw.edu</a>,
 <a href="https://econ.washington.edu/people/shiny-shi"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="photo" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/profile-images/Shiny%20Shi%402x%20%286%29.jpg?h=0a97d481&amp;itok=a-RqePl_" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/shiny-shi" hreflang="en">Shiny Shi</a>,
 <a href="mailto:shiny262@uw.edu">shiny262@uw.edu</a>,
 <a href="https://econ.washington.edu/people/roy-sonn"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Roy" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/roy_sonn.jpg?itok=ftdqHOTL" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/roy-sonn" hreflang="und">Roy Sonn</a>,
 <a href="mailto:roysonn@uw.edu">roysonn@uw.edu</a>,
 <a href="https://econ.washington.edu/people/elliot-spears"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Photo_1" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/profile-images/IMG_1951.jpg?itok=K9uTVrqk" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/elliot-spears" hreflang="und">Elliot Spears</a>,
 <a href="mailto:espear1@uw.edu">espear1@uw.edu</a>,
 <a href="/fields/specific/banking-and-financial-markets">Banking and Financial Markets</a>,
 <a href="/fields/international-finance">International Finance</a>,
 <a href="/fields/macroeconomics">Macroeconomics</a>,
 <a href="https://econ.washington.edu/people/tailai-sun"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="tailai" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/img_8414.jpeg?itok=h-aJZoKI" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/tailai-sun" hreflang="und">Tailai Sun</a>,
 <a href="mailto:tailas@uw.edu">tailas@uw.edu</a>,
 <a href="/fields/financial-economics">Financial Economics</a>,
 <a href="/fields/international-finance">International Finance</a>,
 <a href="/fields/international-trade">International Trade</a>,
 <a href="https://econ.washington.edu/people/chih-lun-sung"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="photo" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/profile-images/Chih-Lun%20Sung.jpg?h=15462622&amp;itok=PpIUcYXS" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/chih-lun-sung" hreflang="en">Chih-Lun Sung</a>,
 <a href="mailto:chihls@uw.edu">chihls@uw.edu</a>,
 <a href="https://econ.washington.edu/people/jacob-tasto"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Jacob" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/jacob_tasto.jpg?itok=60z4jmYz" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/jacob-tasto" hreflang="und">Jacob Tasto</a>,
 <a href="https://econ.washington.edu/people/fatih-turhan"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Fatih Turhan" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/fatih_turhan.jpg?itok=bOefDElM" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/fatih-turhan" hreflang="und">Fatih Turhan</a>,
 <a href="mailto:fturha@uw.edu">fturha@uw.edu</a>,
 <a href="https://econ.washington.edu/people/eric-wang"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="eric" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/eric_wang.jpg?itok=Ru2retbp" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/eric-wang" hreflang="und">Eric Wang</a>,
 <a href="https://econ.washington.edu/people/haohui-wang"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Haohui Wang" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/haohui_wang.jpg?itok=_06LDu5K" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/haohui-wang" hreflang="und">Haohui Wang</a>,
 <a href="mailto:haohuiw@uw.edu">haohuiw@uw.edu</a>,
 <a href="/people/seyoung-won" hreflang="und">Seyoung Won</a>,
 <a href="mailto:seyoung1@uw.edu">seyoung1@uw.edu</a>,
 <a href="/fields/time-series">Time Series</a>,
 <a href="/people/gaoqian-xu" hreflang="und">Gaoqian Xu</a>,
 <a href="mailto:gx8@uw.edu">gx8@uw.edu</a>,
 <a href="https://econ.washington.edu/people/swenny-xue"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Photo" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/profile-images/Swenny%20Yan%20Xue.JPG?itok=wIPyrT_n" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/swenny-xue" hreflang="en">Swenny Xue</a>,
 <a href="mailto:swennyx@uw.edu">swennyx@uw.edu</a>,
 <a href="https://econ.washington.edu/people/linda-yang"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="lindayang" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/profile-images/WechatIMG190.jpg?h=aa9f8b28&amp;itok=0VeJTVnS" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/linda-yang" hreflang="en">Linda Yang</a>,
 <a href="mailto:xyang58@uw.edu">xyang58@uw.edu</a>,
 <a href="https://econ.washington.edu/people/tianzhou-yao"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Tianzhou Yao" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/tianzhou_yao.jpg?itok=gJTmEpFp" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/tianzhou-yao" hreflang="und">Tianzhou Yao</a>,
 <a href="mailto:tianzyao@uw.edu">tianzyao@uw.edu</a>,
 <a href="https://econ.washington.edu/people/dhongkyu-yoon"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Dhongkyu Yoon" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/dhongkyu_yoon.jpg?h=fb53d0d2&amp;itok=oFOMIuvR" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/dhongkyu-yoon" hreflang="und">Dhongkyu Yoon</a>,
 <a href="mailto:dhongkyu@uw.edu">dhongkyu@uw.edu</a>,
 <a href="https://econ.washington.edu/people/bocheng-zhang"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Headshot of Bocheng Zhang" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/img_0701_0.jpeg?h=abbb37dc&amp;itok=76SNHfOY" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/bocheng-zhang" hreflang="und">Bocheng Zhang</a>,
 <a href="mailto:zhan0908@uw.edu">zhan0908@uw.edu</a>,
 <a href="/fields/development-economics">Development Economics</a>,
 <a href="/fields/education">Education</a>,
 <a href="/fields/health">Health</a>,
 <a href="/fields/social-networks">Social Networks</a>,
 <a href="https://econ.washington.edu/people/jinwei-will-zhang"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Will Zhang" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/photos/jinwei_will_zhang.jpg?itok=mmwQwNR2" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/jinwei-will-zhang" hreflang="und">Jinwei (Will) Zhang</a>,
 <a href="mailto:liwzhang@uw.edu">liwzhang@uw.edu</a>,
 <a href="https://econ.washington.edu/people/olivia-zhang"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="Olivia Zhang" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/profile-images/23.08%20%5B1%5D.jpg?h=c98f1be5&amp;itok=cGU9p3DB" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/olivia-zhang" hreflang="en">Olivia Zhang</a>,
 <a href="mailto:zzqq02@uw.edu">zzqq02@uw.edu</a>,
 <a href="https://econ.washington.edu/people/wenping-zhang"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="1" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/profile-images/IMG_3300.JPG?h=c35e2e38&amp;itok=Rb-vKJ1F" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/wenping-zhang" hreflang="und">Wenping Zhang</a>,
 <a href="mailto:wenping@uw.edu">wenping@uw.edu</a>,
 <a href="https://econ.washington.edu/people/haijing-zong"><article>
 <div class="field-wrapper field field-media--field-media-image field-name-field-media-image field-type-image field-label-hidden">
 <div class="field-items">
 <div class="field-item"> <img alt="haijing zong" height="125" loading="lazy" src="/sites/econ/files/styles/portrait_small/public/profile-images/222.jpg?h=1754517a&amp;itok=tM8-Ty2c" width="100"/>
 </div>
 </div>
 </div>
 </article>
 </a>,
 <a href="/people/haijing-zong" hreflang="und">Haijing Zong</a>,
 <a href="mailto:zhaijing@uw.edu">zhaijing@uw.edu</a>,
 <a href="/fields/development-economics">Development Economics</a>,
 <a href="/fields/industrial-organization">Industrial Organization</a>,
 <a href="/fields/labor-economics">Labor Economics</a>,
 <a class="menu-item--active-trail" data-drupal-link-system-path="people" href="/people">People</a>,
 <a data-drupal-link-system-path="people/faculty" href="/people/faculty" title="">Faculty</a>,
 <a data-drupal-link-system-path="node/732" href="/visiting-scholars">Visitors</a>,
 <a data-drupal-link-system-path="people/staff" href="/people/staff" title="">Staff</a>,
 <a aria-current="page" class="is-active" data-drupal-link-system-path="people/graduate-student" href="/people/graduate-student" title="">Graduate Students</a>,
 <a aria-current="page" class="is-active" data-drupal-link-system-path="people/graduate-student" href="/people/graduate-student" title="">All Graduate Students</a>,
 <a data-drupal-link-system-path="node/1057" href="/people/graduate-student/first-year-phd-student">First Year PhD Students</a>,
 <a data-drupal-link-system-path="node/1058" href="/people/graduate-student/job-market-candidate">Job Market Candidates</a>,
 <a data-drupal-link-system-path="node/644" href="/alumni">Alumni</a>,
 <a class="button support-button" href="/support-us" id="giving-link-footer"><span class="fa fa-heart-o fa-2x"> </span> Support Economics</a>,
 <a href="https://www.facebook.com/uwecon"><span class="fa-container"><span class="fa fa-facebook fa-2x"> </span></span><span class="hidden-for-small-only">Facebook</span></a>,
 <a href="https://twitter.com/UWeco"><span class="fa-container"><span class="fa fa-twitter fa-2x"> </span></span><span class="hidden-for-small-only">Twitter</span></a>,
 <a href="https://www.linkedin.com/groups?home=&amp;amp%3Bgid=4410788&amp;amp%3Btrk=anet_ug_hm"><span class="fa-container"><span class="fa fa-linkedin fa-2x"> </span></span><span class="hidden-for-small-only">LinkedIn</span></a>,
 <a href="https://www.instagram.com/uwecon"><span class="fa-container"><span class="fa fa-instagram fa-2x"> </span></span><span class="hidden-for-small-only">Instagram</span></a>,
 <a href="/mailing-list"><span class="fa-container"><span class="fa fa-envelope-o fa-2x"> </span></span><span class="hidden-for-small-only">Newsletter</span></a>,
 <a href="mailto:econdept@uw.edu">econdept@uw.edu</a>,
 <a href="http://www.washington.edu/online/privacy">Privacy</a>,
 <a href="http://www.washington.edu/online/terms">Terms</a>,
 <a href="/sitemap">Site Map</a>,
 <a href="/alumni-update">Alumni Update</a>,
 <a href="/contact">Contact Us</a>,
 <a href="http://uw.edu">University of Washington</a>,
 <a class="exit-off-canvas"></a>]

Cutting Down to Useful Links

url_pattern=re.compile('https://econ.washington.edu/people/[a-z]+\-[a-z]+')
links = [x['href'] for x in r1_bs.find_all('a', {'href': url_pattern})]
links[:5]

['https://econ.washington.edu/people/amre-abken',
 'https://econ.washington.edu/people/afsana-adiba',
 'https://econ.washington.edu/people/shabab-ahmed',
 'https://econ.washington.edu/people/ishmam-al-quddus',
 'https://econ.washington.edu/people/alireza-aminkhaki']

Step 2

Note that the useful data is stored in <div> elements with consistently formatted class names:

r2 = requests.get('https://econ.washington.edu/people/amre-abken')
assert r2.ok
r2_bs = BeautifulSoup(r2.text)

data_dict = {}
field_names = ['email', 'office', 'office-hours', 'biography']
for field_name in field_names:
    search_crit = {'class': re.compile(f'field-name-field-{field_name}')}
    search_obj = r2_bs.find_all('div', search_crit)
    if len(search_obj) > 0:
        data_dict[field_name] = search_obj[0].text.strip()
data_dict

{'email': 'abken@uw.edu',
 'office': 'Savery Hall 319F',
 'office-hours': 'Office Hours\n\nMonday 330-430pm; Wednesday 330-430pm',
 'biography': 'Nazarbayev University'}

`DataFrame`

dfs = []
for url in links[:5]:
    r2 = requests.get(url)
    assert r2.ok
    r2_bs = BeautifulSoup(r2.text)

    data_dict = {}
    field_names = ['email', 'office', 'office-hours', 'biography']
    for field_name in field_names:
        search_crit = {'class': re.compile(f'field-name-field-{field_name}')}
        search_obj = r2_bs.find_all('div', search_crit)
        if len(search_obj) > 0:
            data_dict[field_name] = search_obj[0].text.strip()
    name = re.findall('(?<=https://econ.washington.edu/people/).+', url)[0]
    dfs.append(pd.DataFrame(data = data_dict, index=[name]))
    time.sleep(3)
print(pd.concat(dfs))

`DataFrame`

                             email            office  \
amre-abken            abken@uw.edu  Savery Hall 319F   
afsana-adiba                   NaN               NaN   
shabab-ahmed       sahmed95@uw.edu       Savery 319H   
ishmam-al-quddus               NaN               NaN   
alireza-aminkhaki    aamink@uw.edu               403   

                                                        office-hours  \
amre-abken         Office Hours\n\nMonday 330-430pm; Wednesday 33...   
afsana-adiba                                                     NaN   
shabab-ahmed                                                     NaN   
ishmam-al-quddus                                                 NaN   
alireza-aminkhaki                                                NaN   

                                         biography  
amre-abken                   Nazarbayev University  
afsana-adiba                                   NaN  
shabab-ahmed                                   NaN  
ishmam-al-quddus                               NaN  
alireza-aminkhaki  Sharif University of Technology

Aside: Email Addresses

This is exactly why you should think hard about putting your email address anywhere on a page – they’re extremely easy to extract and spam.

Cookies

Cookies with `requests`

We can save cookies from a request with requests.session():

s = requests.session()
google_r = s.get('https://www.google.com')
google_r.ok

True

Look at Cookies

[x for x in s.cookies]

[Cookie(version=0, name='AEC', value='AZ6Zc-WuxfO0PMKlkGl9cpA2EdWYgvz26f1YaqbLoDrfUHduP1ak9IojrOU', port=None, port_specified=False, domain='.google.com', domain_specified=True, domain_initial_dot=True, path='/', path_specified=True, secure=True, expires=1748406199, discard=False, comment=None, comment_url=None, rest={'HttpOnly': None, 'SameSite': 'lax'}, rfc2109=False),
 Cookie(version=0, name='NID', value='519=zEEzf0jMYxe4nBCiiNChfPwfNF-h7l99lJEzAXDIc8lEd4zz1ow4CDOsMQtJ8voSm2Ep4V-HilVB0798_xbgYEY7_L7_5Rjk0PQb4D6IAh9jzm5VbSGRiT_qiiveQi_XfH3Z8yVLwL6kMHkKPzVKSbxJktdbpaHDdzfT7IGkNbsGNeIYl7PmGDCh9TjENDZivHdqsywr', port=None, port_specified=False, domain='.google.com', domain_specified=True, domain_initial_dot=True, path='/', path_specified=True, secure=False, expires=1748665399, discard=False, comment=None, comment_url=None, rest={'HttpOnly': None}, rfc2109=False)]

Passing These Cookies

You could now use these cookies in a new request with

requests.get(<url>, cookies=s.cookies)

A common use case is authenticating, and then passing the cookies resulting from the authentication to future requests.

Headless Browsers

Use Cases

Sometimes if a page is sufficiently complicated (think Amazon), you can use a headless browser to scrape a page
This is essentially giving a browser instructions on what to do
- First, click this button, then type this text in this field, etc.
Not obvious that it’s super valuable in my experience
If interested, look at Selenium

Web Scraping

Learning Objectives

Ethics

Is Web Scraping Legal?

Is Web Scraping Ethical?

Basics

What is Web Scraping?

Basic Goal

requests

status_code

ok

Content

HTML

HTML Structure

This is a heading

HTML Tables

BeautifulSoup

prettify()

Searching in the HTML

Finding table in BeautifulSoup

Making this Usable

Getting the Headers

Converting BeautifulSoup to List

Making Headers Prettier

Exercise: Getting Table Contents

Solutions: Getting Table Contents

Putting It Together

Faster Solution (If Possible)

When Does pd.read_html Work?

Harder Scraping Problem

Baseball Reference

Find “Advanced Batting”

Use pandas

id tag

Search the Raw HTML

Getting Tables in Comments

Get The Table We Care About

Use pd.read_html

Baseball Reference is Tricky

Using APIs

Data Population on Page

Western States

Western States Results

Network Tab in Chrome

Network Tab json Request

Requesting Directly

Passing json to DataFrame

Scraping Multiple Pages

Data Stored on Multiple Pages

Scraping Multiple Years of WS Results

Plugging Into API

Plugging Into API

Getting Event IDs

Getting the Elements

Putting Together

Authentication

Problem: Sites Recognize Bots

Examples

Potential Solution: Authenticate

Issue

Very Simple – Olympus

Exercise: Olympus

Olympus Headers

Olympus Payload

Constructing the Request

Exercise: Extracting Links

Solutions: Extracting Links

UW Economics Database

Goal (Scraping Non-Tabular Data)

Exercise: Strategy

Approach

Step 1

Cutting Down to Useful Links

Step 2

DataFrame

DataFrame

Aside: Email Addresses

Cookies

What is a Cookie?

Cookies with requests

`requests`

`status_code`

`ok`

`BeautifulSoup`

`prettify()`

Finding `table` in `BeautifulSoup`

Converting `BeautifulSoup` to List

When Does `pd.read_html` Work?

Use `pandas`

`id` tag

Use `pd.read_html`

Passing `json` to `DataFrame`

`DataFrame`

`DataFrame`

Cookies with `requests`