Project 04: JSON and Webcrawler

The first part of this project focuses on implementing a class to parse and dump (i.e. read and write) JSON using recursion.

What is JSON?

JSON (JavaScript Object Notation) is a text-based format used to transmit data. It’s frequently used due to its easy-to-read nature for both humans and machines. For more information and a visualization of what a JSON object looks like, you can check out this link.

There are three main JSON types you’ll need to handle: Objects, Lists, and Values. Values consist of five different types: 4 data types Booleans, Doubles, Integers, and Strings, along with a special null value. We have already given you code for classes that you should use to store each of these data types, named appropriately.

Each class implements the JSON interface, so they inherit the method dump and the static method parse, both of which you will be writing in this project.

JSON syntax

JSON values are formatted as the value itself. If the value is a string, the string should also be surrounded by quotations, and any special characters should be physically written out (for example a new line will be written out as the two character string “\n”). Because there are so many of these characters we have given you two methods to easily convert to and from these different formats, which will be discussed later in the guide.

JSON Lists are denoted by brackets that contain some number of JSON types separated by commas. Example: [1,2,3,4,5]

JSON Objects are denoted by curly brackets that contain some number of members separated by commas. Each member consists of a string, formatted in the same way we would a JSONStringValue, followed by a colon followed by a JSON type. Example: {"name":"Caltech","city":"Pasadena","state":"California","zip code":91125}

Whitespace is often used to make larger JSON files more easily readable for humans, and thus is allowed between any pieces of the above syntax. Such whitespace should be ignored when parsing.

Helper Methods and ParsableString Class

We’ve provided you with a JSONHelpers interface, which has the following methods:

• toFormattedString(String s) - Given a String, returns it as formatted for a JSON file.
• tryParseInteger(String s) - Given a String, returns the parsed integer if it’s an integer and null otherwise.
• tryParseDouble(String s) - Given a String, returns the parsed double if the provided value can be parsed as a double (this includes integers) and null otherwise.

We’ve also provided you with a ParsableString class. This will allow you to easily keep track of what index has been parsed up to so far. All methods to alter this index will also automatically account for whitespace for you. This class has the below methods:

• incrementIndex(int i) - Moves the index forwards i characters, adjusting for whitespace
• parseInternalString() - If the index is currently at the start of a String, parses and returns that String, moving the index forwards accordingly, adjusting for whitespace. Throws a JSONParsingException if it’s not at the start of a String.
• getNumber() - If the index is currently at an integer or double, returns the entirety of that number as a string, moving the index forwards accordingly, adjusting for whitespace. Throws a JSONParsingException if it’s not at the start of an integer or double.

Part 1: JSON Implementation

In this part, you will implement two major methods in the JSON interface that can dump and parse JSON elements respectively. Both of these methods must use recursion to accomplish their goals.

In last week’s project you wrote a toString method for HTMLManager and several helper methods for HTMLParser, each of which used stacks and queues. This time, to create JSON’s dump and parse methods you will instead use recursion.

Both dump and parse MUST be written recursively when indicated to receive any credit.

public String dump(int numIndents)

1
2
3
4
5
6
7

[
  "Pasadena",
  {
    "car": 2.5,
    "metro": 5
  }
]

1
2
3
4

[
  JSON element,
  JSON element
]

1
2
3
4

{
  "key": JSON element,
  "key": JSON element
}

public static JSON parse(String inputStr)

If you’re stuck on some failing test cases, below are a couple of hints about potential edge cases for parse().

1
2
3
4
5
6
7
8
9

[
    "a", 
    [
        "cat",
        "dog"
    ],
    "b"
]

1
2
3
4

[
    "cat"
]]

Part 2: WebCrawler

What is a WebCrawler?

A web crawler is a program designed to browse the internet, and index webpages, traveling from page to page using the links on each page. Our web crawler will also be scraping these webpages for their textual content, which will be useful in next week’s project where you will construct a search engine.

Implementation

public static JSONObject crawl(String link, int maxDepth)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

{
  "https://Wikipedia.org/wiki/Chicken":
  {
    "content": "Gallus gallus domesticus",
    "links":
    {
      "https://Wikipedia.org/wiki/Pet": 1,
      "https://Wikipedia.org/wiki/Hemoglobin": 1
    }
  },
  "https://Wikipedia.org/wiki/Pet": 
  {
    "content": "companion animal",
    "links":
    {
      "https://Wikipedia.org/wiki/Chicken": 1,
      "https://Wikipedia.org/wiki/Koi": 1
    }
  },
  "https://Wikipedia.org/wiki/Hemoglobin" : 
  {
    "content": "blood",
    "links":
    {
      "https://Wikipedia.org/wiki/Hyperbolic_functions": 1
    }
  }
}

Helper Methods

Additionally, we have given you two more helper methods in WebCrawler:

• public static String getPageText(String page): given the link for a webpage, this method returns the contents of the webpage in raw HTML. You may want to use this in conjunction with HTMLParser to get the HTMLTags for a link.
• public static boolean ignoreHref(String s): given a potential link (the text following an “href=”), returns true if the contents are not related to a webpage (and should therefore be ignored by crawl).
• public static boolean ignoreLink(String link): given a String formatted as a link, returns true if the contents point to an invalid webpage (and should therefore be ignored by crawl).

These methods will be necessary for your implementation of crawl: after obtaining a Queue of tags, you can process the Queue to find tags that correspond to links. For instance, if the link to webpage B appears in webpage A, then once you get a Queue of tags in webpage A, the tag <a href="link to B"> will be in that Queue. Note that the actual link to B is a substring of the String "<a href="link to B">", which you want to extract using the HREF regular expression.

Result

Congratulations! You’ve successfully created a database of links and webpages using your webcrawler and your own implementation of JSON. If you look at the tests, you will see that in the tests, we dump out the databases you’ve created (using your dump() method). Next week, you will see what we can do with these dumped-out databases.