Building a Ruby CLI Gem From Scratch

jck
Level Up Coding
Published in
10 min readMar 15, 2020

--

As a student in Flatiron School’s Software Engineering program, I was tasked with creating a command line application in Ruby. This application would be bundled into a nice neat package readily available to implement in another programmers’ application and is referred to in Ruby as a “gem”. My gem, called TopNewsD1, scrapes articles from ESPN’s NCAA Men’s Basketball homepage via Nokogiri and prints them into your terminal, and will update in real time based on the live website. My gem was built based on the given specs:

1. Provide a CLI (Command Line Interface)

2. Provide access to data from a given webpage

3. The data provided must be able to go one level deep (A “level” is where a user can make a choice and then get detailed information about their choice)

This article will give a generalized step-by-step walkthrough of how I built this gem, and what my gem does, as well as my thoughts on the experience of building the gem.

Step One — Create the Repository

If you are reading this article with intentions of building your own gem, there are many resources online that can provide supplemental information on this specific step. For my purposes, I used bundler to create my gem repository and all the files necessary to have this gem work properly. In my terminal I navigated to my desired location and typed bundle gem TopNewsD1 into my terminal and voila, everything needed to begin properly was created.

Bundle Gem in the Terminal

The only files above that were created that I needed to edit were the gemspec, and the README. The files themselves will actually tell you what you need to edit, save for adding a few development dependencies to the gemspec.

Gemspec Dependencies

At the bottom of the gemspec I had to add a development dependencies for bundler, rake, rspec, and pry, and a general dependency for nokogiri so that any of my files that used scraping could properly scrape the desired webpage.
(After this you will need to connect this directory on your local machine with a repository on GitHub. You can check out GitHub’s article for specifics on this)

Step Two — Operation and Attributes

I already knew what I wanted my gem to scrape ESPN’s NCAA Men’s Basketball homepage and provide the articles based on what the website showed at the time, but I needed to decide how I wanted my application to operate. I used some of the resources provided by Flatiron to give me an idea of how this would be done.
First and foremost, I needed a way to run my program. This is done by objects calling on other objects in somewhat of a chain. The first link in the chain is located in the bin folder under a file I created called TopNewsD1.

TopNewsD1 in the bin Folder

There are only four lines of actual code but they are all essential to operating. The most important piece of code here is require “TopNewsD1”. This line requires the TopNewsD1 file that will be our environment for the whole application (I also had to add these same lines of code to the file console in the bin folder).

Let’s dive into our environment a little bit.

My environment file is the file that everything in the application hinges upon.

Environment Setup

Because I am using scraping I needed open-uri & nokogiri. These are required on the first two lines. I will also undoubtedly run into problems, and will use pry to debug. Lastly I will need to require my files inside our TopNewsD1 folder inside of my lib folder. This is done on lines 5–7. Simple yet powerful, my environment file will let all the files that need to communicate do so.

Back to the file in the bin folder, the last line reads TopNewsD1::CLI.new.call. This line accesses the TopNewsD1::CLI class, creates a new instance, and calls the call method (we will define this class and its methods in a moment). This method call, will be the first object that is called in the chain.

But before we get there we need to back track a little bit, away from writing code back to the basics of my program. I created an outline and gave some general guidelines for myself to proceed. Ultimately, I had to ask myself a very important question:

What defines an article?

An article has a title.

An article has an author.

An article has a description or topic.

An article has a timestamp based on when it was written.

An article has a URL.

This gave me some important information. I knew I needed to get a title, an author, a description or topic, a timestamp, and a URL for each article that I scraped. These will be my attributes to be set.

Step Three — Creating the CLI

Inside of the lib folder, and even deeper into the TopNewsD1 folder I created a file called cli.rb. This file will house all of the necessary code for our user to interact with our application. In the last step, inside of the TopNewsD1 file in the bin folder I called the call method from the class TopNewsD1::CLI.
It’s actually a very simple method and is the next step in the chain.

Call Method

The call method, a measly two lines of code, will list the stories scraped, and give the user a menu to interact with. The list, list_stories, is another method I will define.

List Method

This method outputs a new line followed by “Top Stories:”. The next lines sets the instance variable @stories to a method inside the ::Stories class called today (which I will define later). After that, it takes that instance method and iterates over each item. I used each.with_index(1) to set the first item to 1 instead of 0. This iteration will output a string that is populated from the Stories class with the attributes we mentioned earlier and that are set in that class. This is where i set my structure for my list. It looks something like “1. ‘Title’ by Author — x days/hours ago”.

The next method called in the call method is menu. This is the most complex method in the entire application.

Menu Method

This method is responsible for receiving the input from the user and deciding what to do based on that input. As a default I have set the input equal to nil. The meat of the menu method is a conditional statement that starts with code that says “while the input does not equal “exit”, do this:” and proceeds to output a new line and a string that says “Enter the number of the article you would like to see, type list to see the list again, or type exit to leave.” This is output to the terminal after every interaction so that our user is never confused as to what to do next.

After setting the input to a gets method, I needed to set the boundaries for human interaction. I said “if a humans input is greater than 0 but less than 3 (1 or 2) do this:” (I used this specific definition because any larger number of available inputs or articles to be scraped caused unnecessary complications to the guidelines of my project). Next I set the variable the_story equal to the @stories instance method, with the parameter of “convert given input to an integer, minus one”. This is because of the earlier each.with_index(1) iteration I did setting the first item to 1 instead of 0. The method then outputs a new line, a tab over, then the article chosen by the user in same format as the list. Then it outputs another new line and another tab, followed by the topic OR description based on the article (some articles have generalized topics, while some had a few excerpt lines of the article).

The next lines are another conditional statement that reads “if the article chosen has a URL, do this:”. This block is the first example of nokogiri and open-uri in the application. Inside this block I set a local variable doc equal to the URL of the chosen story. Then I set the variable story equal to the location of the text in the webpage that contained the actual article. The block then outputs a new line, the story, another new line, and the articles URL. In this conditional I set a default response that outputs the ESPN NCAA Men’s Basketball homepage URL if no story URL is present. After this I have a few elsif statements and lastly an else statement. The first elsif statement says that “if input is equal to ‘list’, call the method list_stories again”. This shows the list of stories again. The second elsif statement says that “if input is equal to ‘exit’, call the method goodbye” which I will define momentarily. Lastly our menu method has a simple else statement called that outputs “Sorry, I’m not sure what you are asking.” if the given input does not fall under one of our set definitions.

Our last method in the ::CLI class is a one line goodbye method, which outputs a farewell message for our users that are finished with out application. I’ve now built the whole CLI, but it’s all useless text without its sibling class TopNewsD1::Stories.

Step Four — Build the Scraper

My scraper, TopNewsD1::Stories, contains the final links of the chain. For starters, I need to set the attributes, and the webpage that I will scrape for these attributes.

attr_accessor and doc Instance Variable

As discussed earlier, I need a title, an author, a description or topic, a timestamp, and a URL. I used attr_accessors to set these attributes. Next I used nokogiri and open-uri to set the instance variable @doc to ESPN’s NCAA Men’s Basketball homepage.

After this, I define our today method that I called in the ::CLI class. This method calls another method self.scrape_stories . The self.scrape_stories method sets stories to an empty array that will be pushed upon, and then pushes the stories I will scrape into this array. Finally, the method returns the array that has been populated.

Scraper and Calling the Scraper

The final two methods in the ::Stories class are the actual scraping methods that use nokogiri, open-uri, and the instance variable @doc to scrape the attributes from the webpage.

HTML Element Scrape

These methods first set a variable story to a new instance of ::Stories, then sets each of the attributes equal to their HTML location in the webpage. Because some of these attributes are always present and some are not, default outputs have been set for a few of them. You will also notice that _headline has a description attribute and _story_one has a topic attribute. This is because the headline on the site always has a short description and the first story listed has a generalized topic.

Working Gem Example

List and Menu
Input and Output
Request List Again
Second Story Output
Unrecognized Output
Farewell Message

The code pictured is accurate as of March 14, 2020.

If you’d like to see my updated work, check out my repository.

Conclusion and Experience

In my experience at Flatiron thus far I have had a few ups and downs. From feeling on top of the world after completing a particularly difficult lab (tic-tac-toe, I’m looking at you…) to losing all hope in myself after being stuck for hours (once again, tic-tac-toe). That being said I’ve always learned from it and moved forward with ambition and focus. My problem however is that none of these labs really made me feel like a true programmer; that I had actually built something. I felt like my hand had been held and I was guided through whatever I needed to do. This was and is necessary, but still I was hungry for more.
This gem project was just that. Equivalent to a mother bird pushing her baby out of a nest and saying “fly”, I’ve truly never been challenged in my programming career as much as this project challenged me. I feel like a true member of the software engineering community… albeit a very immature and inexperienced member, but a member nonetheless. There was a lot more that went into this project than what I was able to portray in this article. What is presented as my minimum viable product is actually my third draft of the gem, with the two pretesessors having been scrapped after a scope that was far to ambitious for my abilities, but each time I started from scratch and took what I learned from my failure and grew and eventually turned it into a success. Looking back on the process of building this gem project I am renewed in my confidence to tackle projects and do meaningful work in my future, and hope to one day soon return to my scrapped projects and look at them as elementary.

--

--