Creating Twitch Highlights and Posting Daily to Reddit with Python, Docker and AWS ECS Fargate Part 1

The Project

Over the past few years Reddit has grown to become the 15th most visited site in the world and in my home country of Ireland it is the third most visited site and the largest social network. When new video games are made, one of the ways companies market their game is to make a subreddit and try to post as much content as possible to get potential buyers engaged.

Twitch meanwhile, has grown to 11th and is the leading streaming platform by far where even mid level streamers make 6 figures a year. Gathering a following can lead to a huge financial windfall. Again, one of the ways streamers achieve this is by making a subreddit and try to get viewers engaged.

In this tutorial, we will make a Python script to automatically post the top Twitch clips from a time period from either a game or a Twitch channel to a subreddit.

Pre-requisites — Part 1

• Go to the Twitch API and get a client id.
• Create a Streamable account.
• Create a Reddit account.

Pre-requisites — Part 2

• Install the AWS Command Line Interface (CLI).
• Sign up for a free AWS account.
• Install Docker.

Outline

There are 6 steps to this project:

• Getting the clips.
• Concatenating the clips into one video.
• Posting the video to Reddit.
• Running the code on Docker.
• Pushing the Docker container to AWS Elastic Container Registry.
• Running Fargate tasks on a schedule with the container.

I will cover the first 3 steps in this post and the next 3 in part two.

Getting The Clips

In a previous post, I explained how to get trending Twitch clips into an S3 bucket using AWS Lambda, we used the Twitch API for that project and will do so here as well.

The first thing we need to do is export our environment variables.

export number_of_videos=2
export mode=game
export key=chess
export twitch_id=YOUR_TWITCH_API_ID
export period=week

In this example I have decided to get the top 2 clips from the past week from the hit game “chess”. If you want to get a game with a space in the title you need to use “%20” instead of the space, for example, if you wanted “Fall Guys” you would need the following command. This is because “%20” is the HTML code for space.

export key=Fall%20Guys

When we have this code on Fargate we will be able to create different Task Definitions and change the environment variables, so if instead of requesting chess videos, we want to request a channels clips (such as GMHikaru) we would only have to change the mode and key environment variables to the following.

export mode=channel
export key=gmhikaru

Once we have a client id for the Twitch API we will be able to download the trending clips of the day by first sending a request to get the “slug” of the clip. Slug is what Twitch uses for a clips title. It’s possible to get clips for a game or a channel over a period of time or else to get the currently trending clips. For more information checkout the documentation.

Below is an implementation where we get the slugs and then the most popular videos of the time period are retrieved, put in the /tmp/ folder for later and a list of the videos paths are returned.

Editing the clips

Once we have the clips in our /tmp/ folder we need to edit them into one video. This can be achieved with the moviepy library. To install it simply set up your python environment and run.

pip install moviepy

We take in the list of paths from the previous steps and initialise the clips array, we resize all clips to be the same size to prevent issues with concatenation, concatenate, then write the video file and return the save location of the video file.

It can take a long time to render the video, for a 2 minute video it took me 140 seconds to render it and it increases drastically the longer the video. However, when I run this code on AWS with maximum settings it takes only 30 seconds. This is one of the benefits of cloud computing, it makes high end computers available to anyone with a web browser.

Uploading the video

I decided to upload the video to Streamable first and then to Reddit instead of trying to upload straight to Reddit as they have a clean API and it is easier to track how many views our videos are doing with the Streamable dashboard than building our own.

We are going to be working with 2 different APIs here, both require accounts so you need to store your username and password in environmental variables.

export streamable_username=YOUR_STREAMABLE_USERNAME
export streamable_password=YOUR_STREAMABLE_PASSWORD
export reddit_username=YOUR_REDDIT_USERNAME
export reddit_password=YOUR_REDDIT_PASSWORD

Streamable’s Api has a clean python library which we will install and use.

pip install pystreamable

I found that its possible for the upload to hang and leave the program in an infinite loop. This is very bad if it happens on AWS as it could lead to large charges on our account. So I implemented logic to check every ten seconds if the video has uploaded and if it fails to upload it ends the program.

Reddit is a bit more complicated, you need to get a Reddit id and Reddit secret to be able to post to Reddit. You will need to go to https://www.reddit.com/prefs/apps and click “Create an app”. You will then need to fill in a form like this.

and then when you hit create app you should get a screen like this.

The string below personal use script is your id and the string beside secret is your secret.

export reddit_secret=YOUR_REDDIT_SECRET
export reddit_id=YOUR_REDDIT_ID

Reddit’s API also has a python library called Praw which we will use.

pip install praw

Praw also requires a user agent, we will use the python library “fake user agent” to generate a unique one each time it is run as getting caught automating on Reddit can lead to a ban.

pip install fake-useragent

The final thing we need to set before writing the script is the sub-reddit we are posting on. This is another environment variable that we will be editing in the different task definitions, but for now, leave it as league of legends.

export subreddit=chess

The very final thing we want to do is make a simple main to run these methods.

Thank you for reading this far, in part 2 I cover the process of running the code on a schedule in ECS Fargate.

If you found this article useful feel free to connect with me on LinkedIn.