WebScraping-Day2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3e89c5b3",
   "metadata": {},
   "source": [
    "<img align=\"left\" src=\"https://ithaka-labs.s3.amazonaws.com/static-files/images/tdm/tdmdocs/tapi-logo-small.png\" />\n",
    "\n",
    "This notebook free for educational reuse under [Creative Commons CC BY License](https://creativecommons.org/licenses/by/4.0/).\n",
    "\n",
    "Created by [Elizabeth Wickes](https://ischool.illinois.edu/people/elizabeth-wickeshttps://ischool.illinois.edu/people/elizabeth-wickes) for the 2023 Text Analysis Pedagogy Institute, with support from [Constellate](https://constellate.org).\n",
    "\n",
    "For questions/comments/improvements, email wickes1@illinois.edu.<br />\n",
    "____"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68f932d1",
   "metadata": {},
   "source": [
    "# Web Scraping Toolkit 2\n",
    "\n",
    "This is lesson 2 of 3 in the educational series on `Web Scraping`. This notebook is intended to teach the core problem solving perspectives and tools for webscraping. \n",
    "\n",
    "**Audience:** `Teachers` / `Learners` / `Researchers`\n",
    "\n",
    "**Use case:** `Tutorial` / `How-To` \n",
    "\n",
    "**Difficulty:** Intermediate\n",
    "\n",
    "**Completion time:** 90 minutes\n",
    "\n",
    "**Knowledge Required:** \n",
    "\n",
    "* Python basics (variables, flow control, functions, lists)\n",
    "* Basic file operations (open, close, read, write)\n",
    "\n",
    "**Knowledge Recommended:**\n",
    "\n",
    "* basic html/websites\n",
    "\n",
    "**Learning Objectives:**\n",
    "After this lesson, learners will be able to:\n",
    "\n",
    "1. Learn how to use pathlib and requests.\n",
    "2. Handle delays in a crawler.\n",
    "3. Generate URLs and download files.\n",
    "\n",
    "**Research Pipeline:**\n",
    "\n",
    "1. You have a research question and data in mind.\n",
    "2. You've found some data you want to use.\n",
    "2. **The data is on a website somewhere and you want to get it off the site and into a data file.**\n",
    "3. You do your analysis or other data prep!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "157c0555",
   "metadata": {},
   "source": [
    "# Required Python Libraries\n",
    "\n",
    "* `requests` for downloading things\n",
    "\n",
    "## Install Required Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e8a220f5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: requests in /Users/wickes1/anaconda3/lib/python3.10/site-packages (2.28.1)\n",
      "Requirement already satisfied: charset-normalizer<3,>=2 in /Users/wickes1/anaconda3/lib/python3.10/site-packages (from requests) (2.0.4)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /Users/wickes1/anaconda3/lib/python3.10/site-packages (from requests) (3.4)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /Users/wickes1/anaconda3/lib/python3.10/site-packages (from requests) (1.26.14)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /Users/wickes1/anaconda3/lib/python3.10/site-packages (from requests) (2022.12.7)\n",
      "Requirement already satisfied: lxml in /Users/wickes1/anaconda3/lib/python3.10/site-packages (4.9.1)\n",
      "ERROR: unknown command \"instal\" - maybe you meant \"install\"\n"
     ]
    }
   ],
   "source": [
    "### Install Libraries ###\n",
    "\n",
    "# Using !pip installs\n",
    "!pip install requests\n",
    "!pip install lxml\n",
    "!pip instal bs4\n",
    "\n",
    "# Using %%bash magic with apt-get and yes prompt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "5480e2a8",
   "metadata": {},
   "outputs": [],
   "source": [
    "### Import Libraries ###\n",
    "\n",
    "#3rd party\n",
    "import requests\n",
    "from lxml import etree\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "import re\n",
    "import time\n",
    "import pathlib\n",
    "import csv"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48d49cf1-257f-48fe-bb48-179e5894c69c",
   "metadata": {},
   "source": [
    "# Data management and file organization"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65b6ab18-9934-4b28-9d2f-37623f8ac9e1",
   "metadata": {},
   "source": [
    "Some diagrams pulled from previous work: https://www.ideals.illinois.edu/items/93987\n",
    "\n",
    "Some themes for organizing your data with web scraping:\n",
    "\n",
    "* keep the project in a folder, ideally under version control\n",
    "\t* yes things get weird when you start having thousands of files, but that shouldn't stop you\n",
    "* use folders within that folder to contain things\n",
    "* be consistent with names\n",
    "* put meaningful information in the filenames\n",
    "\t* an ID or other content that will easily connect things back to the original source\n",
    "* retain that meaning between files as you convert them\n",
    "\t* eg use the same identifier information between different versions of the entity \n",
    "\n",
    "## Before we move on, some considerations\n",
    "\n",
    "There are a few things to consider before we move on where we are programmatically downloading things from someone else's server. Not every website wants to be scraped. Some have restrictions some have blocks, and there's a certain kind of etiquette that we want to follow.\n",
    "\n",
    "First, we want to keep the speed we are hitting their server to something reasonable. This is usually a minimum of 4 seconds, but I've worked with pages that asked for 30 seconds delays. \n",
    "\n",
    "Second, some ask that you only do large scale harvesting or scraping during \"off peak\" times. This often means overnight.\n",
    "\n",
    "Third, some pages may just completely ban scraping tools from being used. Usually this is because they have an API they'd prefer you to use (and usually pay for) or because the data is sensitive in some way.  Let's look a few examples. \n",
    "\n",
    "* Linkedin has a hard block on programmatic web scraping because their data is really valuable and they want to sell it to you.\n",
    "* Google will quickly block you from scraping their results because they want to you to use an API. Many of theirs are open and reasonable to use, but they don't want HTML scraping.\n",
    "* Archive of Our Own (AO3) has a block against it because they don't want search engines to index the results. This gives them control over story and author information and the ability to fully take things down as needed. \n",
    "\n",
    "But how can you know for sure? This can be hard and there's no single answer. You can often check the `robots.txt` file for the website. You can read about this file here: https://en.wikipedia.org/wiki/Robots.txt Very generally, it will contain information for humans and for bots, and give you an idea about limitations, etc. Not every site will have it, but most with data will. \n",
    "\n",
    "* https://en.wikipedia.org/robots.txt\n",
    "* https://archiveofourown.org/robots.txt\n",
    "\t* my favorite \"cruel but efficient\"\n",
    "\t* note the crawl delay\n",
    "* https://www.fanfiction.net/robots.txt\n",
    "\n",
    "You can ask for permission to go out of bounds for this, especially for research. Just be respectful.\n",
    "\n",
    "## Handling delays\n",
    "\n",
    "Most programming languages will have some ability to \"delay\"actions. We will use the `time` module in Python to delay our execution.\n",
    "\n",
    "`time.sleep(seconds)` takes a number of seconds and pauses script execution for that long. Other languages use `ms` instead, so be mindful if switching!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "504b8859-6b6d-48c8-9fc0-b259c9ac44f1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hello!\n",
      "hello!\n",
      "hello!\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "for _ in range(3):\n",
    "\tprint(\"hello!\")\n",
    "\ttime.sleep(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9f26eb7-c99c-43df-8e67-37542402c0da",
   "metadata": {},
   "source": [
    "## Downloading things off one page\n",
    "\n",
    "Starting with the simplest version for sure, we have one page with a side of links, and we want to download the results of those links. What those files are, doesn't really matter because you're downloading them to disk. \n",
    "\n",
    "So what I love about this page is that they just have the sql statement right at the top of the page. \n",
    "\n",
    "https://calphotos.berkeley.edu/cgi/img_query?where-taxon=Allium+anceps\n",
    "\n",
    "Let's take a look at the structure here:\n",
    "\n",
    "* clearly these are coming from a database\n",
    "* there are multiple pages\n",
    "* the images are displayed on the page\n",
    "* there are detail links by each image\n",
    "* being displayed in a table\n",
    "\n",
    " Tip: Chrome XPath Helper tool\n",
    "\n",
    "I like to use this to preview the structure of the elements.\n",
    "\n",
    "There are a variety of tools you can use for this part! Our basic goal for this is to get URL for each of the pictures. Once we have those collected, we can run through them to download each. I'm going to provide these URLs for now so we can focus on the downloading. \n",
    "\n",
    "Just a small preview of this xpath we'll be using:\n",
    "\n",
    "`//td//img/@src`\n",
    "\n",
    "* we can use `//img` to get all the images on the page, but most pages will have other images. Best practice is to include something more specific to disambiguate. This is why I have `td` in here.\n",
    "* Using `@src` allows me to request that it return the value for the source property\n",
    "* the URLs for the images appear to have a specific folder structure, which I could have also used to gather them\n",
    "* the URLs gathered are relative links, meaning that I'll need to build the full URL when I'm doing my pass over them. \n",
    "\n",
    "Let's open the text file with the URLs and start building those up. As mentioned, these are relative links so we will need to do a bit of editing to get them into the full pattern.  You can check out a link on the main page to inspect what the full URL should be and what the relative links are. Looking a that we can discover that the \"base\" url should be.\n",
    "\n",
    "Here's a full link:\n",
    "`https://calphotos.berkeley.edu/imgs/128x192/0000_0000/1209/2448.jpeg`\n",
    "\n",
    "And here's the corresponding relative link: \n",
    "\n",
    "`imgs/128x192/0000_0000/1209/2448.jpeg`\n",
    "\n",
    "This means we'll need to prepend `https://calphotos.berkeley.edu` before each URL to have the full one. There are several ways you can do this and this is a great time to practice your core Python skills. \n",
    "\n",
    "Some notes:\n",
    "\n",
    "* using list comprehension syntax here\n",
    "* using `readlines` to read it in, which returns a list of strings, each string is a line from the file plus a newline character\n",
    "* `strip` is needed to take the ending newline character off\n",
    "* I'm concatenating the base before the url from the line, but note that I didn't include the final / because there's already an opening one from the url. \n",
    "* This will result in a list of all the urls."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "66db874e-1afb-4f99-b531-7c7734121214",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('pictures.txt', 'r', encoding = 'utf-8') as infile:\n",
    "    # urls = infile.readlines()\n",
    "    urls = ['https://calphotos.berkeley.edu' + u.strip() for u in infile.readlines()]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78a0f472-37de-4cd6-91cc-68a61bac99b0",
   "metadata": {},
   "source": [
    "## Working with `requests`\n",
    "\n",
    "Let's try something basic!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "103327a2-fe8e-434a-9b86-b1d58bab9db7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<Response [200]>\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "\n",
    "url = \"https://loripsum.net/api/1/plaintext/short\"\n",
    "result = requests.get(url)\n",
    "\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e0f0915-5b6c-4d9a-a254-1b01645814b9",
   "metadata": {},
   "source": [
    "So what we're seeing here is a sucessfull connection, but not the text.  We have to ask about that explicitly from out result object.\n",
    "\n",
    "We do this with `.text` (no parens!) this will allow us to ask for a variable value within out object (versus calling a function). Some objects just work this way, and we know how to do this by looking at the documentation or a tutorial.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "1f9f10bf-768d-4338-ac4b-07017f979379",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b'Lorem ipsum dolor sit amet, consectetur adipiscing elit. Equidem e Cn. Nam Pyrrho, Aristo, Erillus iam diu abiecti. Iam id ipsum absurdum, maximum malum neglegi. Prioris generis est docilitas, memoria; Tria genera bonorum; Duo Reges: constructio interrete. \\n\\n'\n"
     ]
    }
   ],
   "source": [
    "print(result.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76637817-8524-4818-a6a7-d3b33c59d2a9",
   "metadata": {},
   "source": [
    "us to make these objects. \n",
    "\n",
    "We can use the `mkdir()` method to create a folder, and then use the `/` concatenation operator to combine them.\n",
    "\n",
    "`pathlib` has two awesome path object methods to write out content:\n",
    "\n",
    "* `write_text(text stuff)`\n",
    "\t* for text!\n",
    "* `write_bytes(a bytes or non-text doodad)`\n",
    "\t* briefly, for stuff that isn't text\n",
    "\n",
    "https://calphotos.berkeley.edu/robots.txt\n",
    "\n",
    "We have a list of URLs now, so we can loop through those and begin downloading them. There are a few tasks we'll need to accomplish.\n",
    "\n",
    "* create the file name (from the file name)\n",
    "* create a directory for the new files to go into\n",
    "* create the full destination path (target folder plus file name)\n",
    "* open up the requests connection\n",
    "* access and write the content\n",
    "* close the connection\n",
    "* wait for 5 seconds\n",
    "\n",
    "This is a lot and we build it up bit by bit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "73931989-6748-4299-96df-e012fabe8731",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pictures/0903_0732.jpeg\n",
      "pictures/1002_0400.jpeg\n",
      "pictures/1102_0790.jpeg\n",
      "pictures/1102_0792.jpeg\n",
      "pictures/1207_0067.jpeg\n",
      "pictures/1207_0083.jpeg\n",
      "pictures/1207_0084.jpeg\n",
      "pictures/1207_0086.jpeg\n",
      "pictures/0408_1095.jpeg\n",
      "pictures/0608_2437.jpeg\n",
      "pictures/0608_2438.jpeg\n",
      "pictures/0608_2439.jpeg\n",
      "pictures/0608_2440.jpeg\n",
      "pictures/0608_2441.jpeg\n",
      "pictures/0608_2442.jpeg\n",
      "pictures/0608_2443.jpeg\n",
      "pictures/0608_2444.jpeg\n",
      "pictures/0209_0663.jpeg\n",
      "pictures/0209_0664.jpeg\n",
      "pictures/0209_0665.jpeg\n",
      "pictures/0209_0666.jpeg\n",
      "pictures/0209_0667.jpeg\n",
      "pictures/0509_0139.jpeg\n",
      "pictures/1209_2447.jpeg\n",
      "pictures/1209_2448.jpeg\n",
      "pictures/0611_1218.jpeg\n",
      "pictures/0611_1219.jpeg\n",
      "pictures/0611_1220.jpeg\n",
      "pictures/0611_1221.jpeg\n",
      "pictures/0611_1222.jpeg\n",
      "pictures/0611_1223.jpeg\n",
      "pictures/0413_3699.jpeg\n",
      "pictures/1113_3030.jpeg\n",
      "pictures/1115_2820.jpeg\n",
      "pictures/1115_2821.jpeg\n",
      "pictures/1115_3063.jpeg\n",
      "pictures/1115_3064.jpeg\n",
      "pictures/1017_1587.jpeg\n",
      "pictures/1017_1588.jpeg\n",
      "pictures/1017_1589.jpeg\n",
      "pictures/0918_2740.jpeg\n",
      "pictures/0918_2741.jpeg\n",
      "pictures/0918_2742.jpeg\n",
      "pictures/0918_2743.jpeg\n",
      "pictures/0918_2744.jpeg\n",
      "pictures/0720_2473.jpeg\n",
      "pictures/0720_2474.jpeg\n",
      "pictures/0720_2475.jpeg\n"
     ]
    }
   ],
   "source": [
    "import pathlib\n",
    "import time\n",
    "import requests\n",
    "\n",
    "# create the target folder object\n",
    "target = pathlib.Path('pictures')\n",
    "# make the directory if needed\n",
    "# does nothing if already exists\n",
    "target.mkdir(exist_ok=True)\n",
    "\n",
    "for u in urls:\n",
    "    parts = u.split('/')\n",
    "    last_two = parts[-2:] # grab the last two parts\n",
    "    fname = \"_\".join(last_two)\n",
    "    # print(fname)\n",
    "    p = target / pathlib.Path(fname)\n",
    "    print(p) # this is the full path\n",
    "    r = requests.get(u) #open connection\n",
    "    p.write_bytes(r.content) # get content, write bytes\n",
    "    r.close() # always close your connection!!!\n",
    "    time.sleep(5) # pause to not anger the server"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3925473e-b048-4791-959a-2fcd9df2ef45",
   "metadata": {},
   "source": [
    "One thing I always check at this point is the file size for everything that has downloaded. When in jupyter on a cloud service, that can be hard, but `!` to the rescue."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "757e86f7-5cf5-4891-b977-335f74b2324b",
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls -l pictures"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f17bdc6-d435-47f3-8815-7dd5b441867c",
   "metadata": {},
   "source": [
    "Now, what if we had many or some messed up? Using pathlib is awesome here. We can utilize the `exists()` method to check if the file we are proposing to make already exists. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2dee04f9-c367-4ebb-a576-bae737591c03",
   "metadata": {},
   "outputs": [],
   "source": [
    "target = pathlib.Path('pictures')\n",
    "target.mkdir(exist_ok=True)\n",
    "\n",
    "for u in urls:\n",
    "    parts = u.split('/')\n",
    "    last_two = parts[-2:] # grab the last two parts\n",
    "    fname = \"_\".join(last_two)\n",
    "    p = target / pathlib.Path(fname)\n",
    "    # use .exists to check\n",
    "    if p.exists():\n",
    "        print(\"already done!\")\n",
    "    else:\n",
    "        print(p)\n",
    "        r = requests.get(u)\n",
    "        p.write_bytes(r.content)\n",
    "        r.close()\n",
    "        time.sleep(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1c540aa-6d0d-473d-a03d-3e415277f369",
   "metadata": {},
   "source": [
    "So we've seen how to gather some contents from a web site, download a group of things from a website, pause a scraper, and even check if that file already exists.\n",
    "\n",
    "This may have been about files, but this could also be about pages. \n",
    "\n",
    "\n",
    "# Generating URLs to download pages from\n",
    "\n",
    "Let's look here: https://calphotos.berkeley.edu/flora/\n",
    "\n",
    "https://calphotos.berkeley.edu/flora/sci-A.html\n",
    "\n",
    "Say that we wanted to automatically grab all the URLS for the scientific names. https://calphotos.berkeley.edu/flora/ looking here we can see that the pages all go from A-Z in the URLs. There's other things we can do, we know that it should contain `flora/sci` within the content. We could get all the URLS, filter, and then use that as our list. But let's try generating the URLs.\n",
    "\n",
    "Often times you'll need to generate numbers or other things within a url. You can use a for loop with `range(number)` to generate a set of numbers \n",
    "\n",
    "In this case we have this theme of `base + letter + .html`. No, we don't need to make all of these ourselves.  The `string` module actually has some fun stuff to keep in mind!\n",
    "Note that most of the items in this module are variables that you are importing instead of functions. This just means that there won't be `()` after the names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "757ae9c6-6ba9-428e-aed0-875ac82f8de1",
   "metadata": {},
   "outputs": [],
   "source": [
    "import string\n",
    "\n",
    "print(string.ascii_uppercase)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbfa41ef-fd47-4152-b5a0-3b53ce1d6729",
   "metadata": {},
   "source": [
    "We can see that we have the letters, let's put it in action. Our url looks like this: `https://calphotos.berkeley.edu/flora/sci-A.html` So hopefully you can see where we might put the letter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "83fb300d-1a52-4ab2-9ed8-03ad9b0be1b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "import string\n",
    "\n",
    "\n",
    "for letter in string.ascii_uppercase:\n",
    "    url = \"https://calphotos.berkeley.edu/flora/sci-\" + letter + \".html\"\n",
    "    # print(url)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4bd480f0-b4ac-4b6a-93e0-99ab1ab68c3b",
   "metadata": {},
   "source": [
    "Let's add some code to download and save these pages to disk and in a folder like we did before. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "f5a539d2-5eb6-4fa8-80e8-31d29d3b56e2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://loripsum.net/api/1/plaintext/short\n",
      "https://calphotos.berkeley.edu/flora/sci-A.html\n",
      "https://calphotos.berkeley.edu/flora/sci-B.html\n",
      "https://calphotos.berkeley.edu/flora/sci-C.html\n",
      "https://calphotos.berkeley.edu/flora/sci-D.html\n",
      "https://calphotos.berkeley.edu/flora/sci-E.html\n",
      "https://calphotos.berkeley.edu/flora/sci-F.html\n",
      "https://calphotos.berkeley.edu/flora/sci-G.html\n",
      "https://calphotos.berkeley.edu/flora/sci-H.html\n",
      "https://calphotos.berkeley.edu/flora/sci-I.html\n",
      "https://calphotos.berkeley.edu/flora/sci-J.html\n",
      "https://calphotos.berkeley.edu/flora/sci-K.html\n",
      "https://calphotos.berkeley.edu/flora/sci-L.html\n",
      "https://calphotos.berkeley.edu/flora/sci-M.html\n",
      "https://calphotos.berkeley.edu/flora/sci-N.html\n",
      "https://calphotos.berkeley.edu/flora/sci-O.html\n",
      "https://calphotos.berkeley.edu/flora/sci-P.html\n",
      "https://calphotos.berkeley.edu/flora/sci-Q.html\n",
      "https://calphotos.berkeley.edu/flora/sci-R.html\n",
      "https://calphotos.berkeley.edu/flora/sci-S.html\n",
      "https://calphotos.berkeley.edu/flora/sci-T.html\n",
      "https://calphotos.berkeley.edu/flora/sci-U.html\n",
      "https://calphotos.berkeley.edu/flora/sci-V.html\n",
      "https://calphotos.berkeley.edu/flora/sci-W.html\n",
      "https://calphotos.berkeley.edu/flora/sci-X.html\n",
      "https://calphotos.berkeley.edu/flora/sci-Y.html\n"
     ]
    }
   ],
   "source": [
    "import string\n",
    "import requests\n",
    "import pathlib\n",
    "import time\n",
    "\n",
    "target = pathlib.Path(\"sci-pages\")\n",
    "target.mkdir(exist_ok=True)\n",
    "\n",
    "for letter in string.ascii_uppercase:\n",
    "    url = \"https://calphotos.berkeley.edu/flora/sci-\" + letter + \".html\"\n",
    "    print(url)\n",
    "    p = target / pathlib.Path(\"sci-\" + letter + \".html\")\n",
    "    r = requests.get(url)\n",
    "    p.write_text(r.text)\n",
    "    r.close()\n",
    "    time.sleep(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ed193350-0127-42a0-91f3-162bfa293b87",
   "metadata": {},
   "outputs": [],
   "source": [
    "for p in target.glob(\"*.html\"):\n",
    "    print(p)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7db75971-beec-42c6-98d6-ce2dbbc13262",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "https://www.crummy.com/software/BeautifulSoup/bs4/doc/#bs4.Tag\n",
    "\n",
    "Simple queries are pretty nice here! \n",
    "\n",
    "Our challenge with this structure is the lack of attribute tags. We just have a bunch of table elements that are nested. Now, this makes some things pretty clean. But it can make it hard to select certain things with precision when there are duplicates. \n",
    "\n",
    "We need to take a look at the structure of the website. What is the content we are after? What makes that location unique compared to the other data?\n",
    "\n",
    "There's no one single answer here. You have to look at the contents and it'll be something specific for each project. But here are some areas to consider:\n",
    "\n",
    "* Is the content only in a specific tag? \n",
    "\t* like an image or h1 text\n",
    "* Is the content in a unique part of the tree?\n",
    "\t* like an image but only the images within a table\n",
    "* Does the content have a specific style class label or html tag attribute that you can latch on to?\n",
    "\t* like the p tags that are marked for formatting with `class = \"name\"`\n",
    "\t* This may me semantic, like a name, or something structural that uniquely identifies what you need\n",
    "* Does the text content within the tag have something unique you can check for?\n",
    "\t* like you want all the `td` cells inside a `tr` where the first `td` cell starts with \"Total:\". Effectively, you want the contents of a row where the first bit of text starts with \"total\"\n",
    "* Does an attribute have a specific value that you can check for?\n",
    "\t* like the `href` for an `a` tag has something specific, as in, you want to check all the hyperlinks but only want the ones that link to a specific subdomain\n",
    "\n",
    "Each of these situations can be coded up. You can usually use some combination of the selection/extraction tool itself along with core python. How you divide that up will depend on your skills with the tools and how nicely the content will play with them. \n",
    "\n",
    "### Some caveats about Beautiful Soup\n",
    "I'll be honest, I'm not the biggest fan of using this for extraction. However, the utility is there and for simpler things it's pretty straight forward. We will be looking at xpath queries later, and for highly structured html or more complex queries, it really is much more straight forward. \n",
    "\n",
    "Another consideration: the searching/parsing of Beautiful Soup is generally slower than what lxml can do. Now, for a few dozen or a few hundred files this shouldn't impact you very much. Use whichever clicks the first for your needs. However, should the number of queries go up into the thousands or millions, you'll want to switch over. Speed may not end up mattering because you can get in and out quickly, and don't need to rerun the results. So this isn't a hard rule. Just keep it in mind. \n",
    "\n",
    "## Extracting things with BeautifulSoup\n",
    "\n",
    "You can read the documentation here for bs4: https://www.crummy.com/software/BeautifulSoup/bs4/doc\n",
    "\n",
    "Some of the lingo on this page may not make a ton of sense to you if you haven't spent some intense personal time in the land of XML or metadata. However, this is where librarian instructors can really shine! Many of us are very used to these kinds of discussions, and you can leverage that expertise to promote your workshops.\n",
    "\n",
    "The sum of it is, each tag is like a node in a tree. That node will have some combination of parent, child, sibling, ancestors, and descendants. This is how you navigate a tree. There's lots of examples on their page that you can work through, but the best way to get used to it is just to mess around. \n",
    "\n",
    "## Loading an html file into beautiful soup\n",
    "\n",
    "Let's read in a single file to explore some of these tools. In this code we just want to grab one file to play with from our target folder. This also lets us practice a bit with our core python!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "07a8a0ef-ad90-4cff-85cb-238ea04ef484",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pathlib\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "first_file = next(target.glob('*.html')) # just a fancy way to ask it to iterate once\n",
    "\n",
    "soup = BeautifulSoup(first_file.read_text(), 'html.parser') "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68fe9877-1d3a-48dd-9cef-e0c710d75431",
   "metadata": {},
   "source": [
    "They always use `soup` as the variable name for the parsed content, so I'm using that to match. I generally suggest you do the same with your own work so things match up with documentation. \n",
    "\n",
    "From here we can now operate on `soup` in a variety of ways. \n",
    "\n",
    "### Simplest extraction\n",
    "The simplest query is just to go after all of a single element. Maybe you want all the images on a page or all the links. You can grab those and run filters etc. with their content in regular python if need be. This can be a nice place to start and allows you to avoid some of the query complexity. \n",
    "\n",
    "You can use `soup` with dot notation and a single tag to grab the first one that the parser sees. This will return the entire element."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7333d4df-ed8e-4039-8c3c-a0354858ab1e",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(soup.a)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7ed139d-166c-4296-9829-fd6dd407948c",
   "metadata": {},
   "source": [
    "```html\n",
    "<a href=\"/\">CalPhotos</a>\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f78d7bf7-4ef6-4607-8cac-b1d02b58efe2",
   "metadata": {},
   "source": [
    "\n",
    "Yup, this is the top of the page. First one it sees. Maybe this is the one you want? Maybe not. This can be good if you need to start digging around the tree and the first ones coming up are the ones you want. You can also chain these together to go directly at something, presuming that the first one it sees is the one and only one you want.\n",
    "\n",
    "```python\n",
    "print(soup.body.table.tr.td)\n",
    "```\n",
    "```html\n",
    "<td bgcolor=\"DFE5FA\" width=\"5%\">\n",
    "<!-- uncomment the following line to add a logo ----->\n",
    "<!-- img align = left border=0 height=130 src = \"/calflora/icon.gif\" alt = \"CalFlora\"-->\n",
    "<br/>\n",
    "</td>\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92580475-1184-4169-b8db-6d4a299b4a8a",
   "metadata": {
    "tags": []
   },
   "source": [
    "\n",
    "You can also ask for the content of the tag with the dot notation. \n",
    "\n",
    "```python\n",
    "print(soup.a.contents)\n",
    "\n",
    "['CalPhotos']\n",
    "```\n",
    "\n",
    "And ask for an attribute's value using dictionary-like notation.\n",
    "\n",
    "```python\n",
    "print(soup.a['href'])\n",
    "\n",
    "'/'\n",
    "```\n",
    "(this result does make sense, as it is linking back to the main page but with a relative link)\n",
    "\n",
    "Generally our queries will be more complex than these simple ones, but this core syntax is good to keep in mind because we will use it in conjunction with more complex queries. \n",
    "\n",
    "## Extracting multiple things\n",
    "\n",
    "In our case we want to examine all the link tags, not just the first ones. We can use `find_all`. They also have a specific section on this you can read more about: https://www.crummy.com/software/BeautifulSoup/bs4/doc/#searching-the-tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8bf40026-d5c1-4c96-ab2f-098f2315ecc0",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(soup.find_all('a'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c7ca669-edec-47b8-8e5d-e1277d10da47",
   "metadata": {},
   "source": [
    "This gives a large list of all the things. As mentioned, given that this is all the links there will be extras we don't want. There are two ways we can improve these results: 1) make our extraction query more specific or 2) attempt to filter out the content we don't want. \n",
    "\n",
    "Both are good strategies to consider. You may not be able to uniquely pinpoint the ones you want within the structure and thus need to look at the content itself, or maybe the content all looks the same and you depend on the structure to disambiguate. Maybe there's a combination of the two!\n",
    "\n",
    "Let's print this out in a for loop for better viewing. \n",
    "\n",
    "```python\n",
    "for a in soup.find_all('a'):\n",
    "    print(a)\n",
    "```\n",
    "```html\n",
    "<a href=\"/\">CalPhotos</a>\n",
    "<a href=\"/flora/\">Plants</a>\n",
    "<a href=\"/browse_imgs/plant.html\">Browse Thumbnail Photos of Plants</a>\n",
    "<a href=\"/cgi/img_query?stat=BROWSE&amp;where-genre=Plant&amp;where-taxon=Wachendorfia+paniculata&amp;title_tag=Wachendorfia+paniculata\">Wachendorfia paniculata</a>\n",
    "<a href=\"/cgi/img_query?stat=BROWSE&amp;where-genre=Plant&amp;where-taxon=Wachendorfia+thyrsiflora&amp;title_tag=Wachendorfia+thyrsiflora\">Wachendorfia thyrsiflora</a>\n",
    "(snip)\n",
    "```\n",
    "\n",
    "We can see a few things. The results we want all likely have the hrefs starting with \"/cgi\". We can do this directly by referencing the attribute name (`href` holds the url) and compiling a regular expression. Yes, you can use regular expression fanciness in here but you can also just put in any string to have it try and match that substring. This is also something you could do in core python with string tools. \n",
    "\n",
    "In this case I've told regex that: the start (`^`) of the string should have `/cgi`. \n",
    "\n",
    "\n",
    "```python\n",
    "for a in soup.find_all('a', href=re.compile('^/cgi')):\n",
    "    print(a)\n",
    "```\n",
    "\n",
    "We can add a search into this on the actual contents of the a tag using the `string` argument. \n",
    "\n",
    "```python\n",
    "for a in soup.find_all('a', href=re.compile('^/cgi'), string = re.compile('var')):\n",
    "    print(a)\n",
    "```\n",
    "This will retain our previous filter but also search the name of the plant for \"var\".\n",
    "\n",
    "This function has a lot of power and the documentation provides a ton of detail: https://www.crummy.com/software/BeautifulSoup/bs4/doc/#find-all Caution, if the core python doesn't make sense then the example likely also won't make sense. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d80ba33-b3e1-406b-872a-cf6656aca3bd",
   "metadata": {},
   "source": [
    "## specifying more structure\n",
    "\n",
    "There's lots of ways to fuss over searching the structure in beautiful soup. The nice thing about being able to provide queries with more content is that you don't have to add loops or other complexities in. You can simply say: I want all the `a` tags that are directly under `p` tags. Or something similar. \n",
    "\n",
    "We can do this with some nice shorthand with the `select()` soup method. \n",
    "\n",
    "This method allows you to specific css structure to select things if you want, but you can also specify more tree structure. This is what we'll be using. \n",
    "\n",
    "You can specify multiple tags here and it will look for those tags within the tree.\n",
    "\n",
    "We could say:\n",
    "\n",
    "```python\n",
    "soup.select(\"p a\")\n",
    "```\n",
    "This would find all `a` tags that exist anywhere inside a `p` tag. (xpath equiv: `//p//a`)\n",
    "\n",
    "We can also say:\n",
    "\n",
    "```python\n",
    "soup.select(\"p > a\")\n",
    "```\n",
    "Where the `a` tags must be direct children of `p`. (xpath equiv: `//p/a`)\n",
    "\n",
    "We can also combine these:\n",
    "\n",
    "```python\n",
    "soup.select(\"table p > a\")\n",
    "```\n",
    "\n",
    "Saying to select all `p` elements anywhere inside of a `table`, and then an `a` tag if directly a child of `p`. (xpath equiv: `//table//p/a`)\n",
    "\n",
    "## Handling the results\n",
    "\n",
    "These results all give you a list of tag objects you can further mess with. \n",
    "\n",
    "We can ask for the contents of the tag:\n",
    "\n",
    "```python\n",
    "for a in soup.select(\"table p > a\"):\n",
    "    print(a.text)\n",
    "```\n",
    "This will give us all the species names. Let's note that the species name is the only thing in the hyperlink. We also want the number next to it. To get this we need all the `p` tag text, but we don't want all the `p` tags. We can accomplish this by navigating the tree more: find all the `a` tags we want and then ask for the parent tag's text. (this is weird but more common than you think). xpath equiv: `//p/a/../text()`\n",
    "\n",
    "```python\n",
    "for a in soup.select(\"td > p > a\"):\n",
    "    print(a.parent.text)\n",
    "```\n",
    "\n",
    "`a.parent` is a relative lookup and \"becomes\" the `p` tags we want. Then `text` is applied to that. \n",
    "\n",
    "We don't want `a.parent.contents` because that will also return the full `a` tag object along with the number. Using `.text` allows us to ask just for the text that is displayed from that element. \n",
    "\n",
    "Looking further at the results we can also notice that we have \"flattened\" this table to just a single column of data. This is because we are ignoring the structure of the table and just grabbing all the individual elements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e1b6ba3-fdad-493a-a9bf-51370ea626ca",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "eca42822",
   "metadata": {},
   "source": [
    "# Exercises (Optional)\n",
    "\n",
    "`If possible, include practice exercises for users to do on their own. These may have clear solutions or be more open-ended.`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "11131305-7648-49f2-93be-bd3326bc4694",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "e42c9736",
   "metadata": {},
   "source": [
    "# Solutions (Optional)\n",
    "`Offer some possible solutions for the practice exercises.`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "227a6d92",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "d2368107",
   "metadata": {},
   "source": [
    "# References (Optional)\n",
    "No citations required but include this if you have cited academic sources. Use whatever format you like, just be consistent. Markdown footnotes are not well-supported in notebooks.[$^{1}$](#1) I suggest using an anchor link with plain html as shown.[$^{2}$](#2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4c7fd8c",
   "metadata": {},
   "source": [
    "1. <a id=\"1\"></a> Here is an anchor link footnote.\n",
    "2. <a id=\"2\"></a> D'Ignazio, Catherine and Lauren F. Klein. [*Data Feminism*](https://mitpress.mit.edu/books/data-feminism). MIT Press, 2020."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}