Breeder's Cup

Leveraged Technologies

NodeJS Web Application
Python, Node, Javascript and HTML Programming Languages
Recurrent Convolutional Neural Network (RCNN)
Recurrent Neural Network (RNN)
DarkNet, Tensorflow, and DarkFlow
ImageNet, YOLO V2, and YOLO V3

Platform Integration

Core Capabilities

We Enhanced The Breeders Cup Online Experience

“Advancements in object recognition technology have led to a new virtual reality viewing experience that could shakeup the dynamics of horse wagering…Rather than functioning as a straightforward VR live stream, an AI and machine learning cocktail on the backend of the broadcast will give viewers real-time information about the horses as they see them walk through the track paddock. Data such as race statistics, betting odds, jockey information and career highlights will pop up into the viewer’s screen, while visuals like interviews, social media feeds and live odds display in the background.”

– Natalie Gagliordi for Between The Lines

Objective

Enhance the experience of Breeder’s Cup attendees and improve the betting experience, drawing in a younger audience.

Solution

A custom web application that displayed racehorse bio information and up to date betting statistics over the real-time 360-degree Breeder’s Cup video stream.

Seisan designed and developed a custom web application that displayed racehorse bio information and up to date betting statistics over the real-time 360-degree Breeder’s Cup video stream. To accomplish this, Seisan used machine learning to consume existing data and “teach” the computer to identify a race horse and its saddle towel to pull the latest information from the Jockey Club’s Equibase.

Project Overview

The final Breeder’s Cup project is a NodeJS web app coupled with a virtual reality experience. Seisan was able to create a machine learning model that trained a computer to identify horses in a targeted stream area. Once positively identified as a horse, our neural model then recognized the saddle towel (appropriate color and number) and pulled the information from the Jockey Club’s Equibase. We used Amazon S3 cloud storage for syncing and building our source data materials and Amazon EFS for securely mapping large network drives to our teams machines while annotating data took place. We also provisioned large GPU cloud servers linked to EFS drives to initiate training jobs.

The project was developed to run locally on event premises to prevent latency and cut out lead time in having to develop, test, and deploy bug fixes or updates based on client feedback. Any changes made could be immediately available to all online viewers at the Breeder’s Cup event. The python application was run on a Nvidia GPU machine and the NodeJS application ran on a Macbook. These machines were networked with the IP 360 Cameras and other machines delivering the 360 virtual reality broadcast.

Technology Details

In order to amass enough data to successfully accomplish machine learning with race horse saddle towels, data was aggregated in a few different ways. Data was also sourced from over 5 terabytes of client and journalized source material. Additional data was crawled from online image sources. In order to overcome our lack of data for the higher number saddle towels, Seisan created virtual renderings using Unity3D of horses and corresponding saddle towels to create image data.

The final training dataset included over 3,000 images to be standardized, labeled, and annotated. Custom scripts were created to process this dataset and format into a manageable file-size. The dataset was handannotated the entire dataset which were stored in PASCAL VOC format to support multiple annotation boxes per image.

Training was accomplished using various RCNN and RNNs. These networks were implemented already in two key machine learning frameworks (DarkNet and a port of DarkNet to TensorFlow, Darkflow). We utilized the fastest image general purpose classifiers (at the time): ImageNet, YOLO V2, and YOLO V3. Refining network parameters and mix/matching color profile datasets, we trained models that performed with over 95% classification accuracy.

A wrapper application was built with NodeJS to: consume classification data, source and cache race-day and equine performance metrics, and provide various visualization views which would be embedded into live streaming software over HTTP.

The central application was a Python script encompassing several responsibilities. The script connected external camera feed via RTMP stream while loading and running trained models to process the frames from this stream. It also provided a visual interface to view the frames with classifications and classification hit-boxes overlayed. Classification timestamps were then pushed to the wrapper application that would marry other cached information to provide real-time data visualization. In addition, the script created an overlay “override” interface that allowed developers to push the correct annotation during infrequent misclassification or video feed interruptions at the event.

Immersive Experience

With the use of VR headset, users were able to explore the backstretch for up close views of Breeders’ Cup contenders, visit the Paddock prerace, and celebrate the race from the Breeders’ Cup Winner’s Circle.

AI & Machine Learning

AI and machine learning gave viewers real-time horse information as they saw them walk through the track paddock. Data such as race statistics, betting odds, jockey information and career highlights populated the viewer’s screen, while visuals like interviews, social media feeds and live odds display in the background.

Results

Seisan’s training datasets performed with greater than 95% classification accuracy in identifying which racehorse was crossing the 360 camera view. Our innovative approach to a 360 livestream had over 100,000 total views, and at its peak, almost 5,000 simultaneous engaged viewers, with well over one million total impressions for the Breeders’ Cup brand.