Making Life Easier Through Machine Learning
I’m going to age myself here, but some of you may remember when you were a kid, having an argument with your friend about some random fact of life. You were 100% sure Gretzky had 215 points in the ’86 season. Your friend was 100% sure it was 208. Arguments ensued and your friendship was over. These days, that scenario can never happen — a simple search on your phone right there would solve it instantly.
My point is technology is making things easier and easier — and that’s a good thing.
As a software developer, I’m always looking to see what I can build to make my life easier. One thing that came up over the holidays was during a game of Mexican Train (Dominoes). After a round is over, and (for the 3rd time in a row) you have a bunch of tiles left, you have to count the “pips” (or nibs, dots, etc). I thought, I’m lazy — why can’t my phone count these tiles for me?
In this post I’ll take you through the steps needed to create an iOS app that can do this counting for you using Turi Create and Core ML. Let’s get started!
You’re only as smart as what you’re taught
The first thing we need to do with machine learning is to train our model. Training a model involves giving the computer examples of what might happen in real life, telling it what the actual answer is, and then allowing the computer to be smarter when it gets an example in the future. The more examples we give it, the smarter it becomes.
Without going into too much detail here, we can train the computer using several different algorithms. Each algorithm specializes with it’s own data, giving it the best output. Image Classification for example, is good for giving it a picture, and having the model tell us what the photo is (ex. Sunset, flower, dog, cat) Drawing classification allows a user to draw something, and the model can tell what it is (ex. Convert a drawn numbers, to the actual value)
For this app, we need to find an object (dots) inside a picture. For this we use an algorithm called Object Detection. Object detection basically comes in 2 steps:
- Localization: Find an area (bounding box) in an image where your object(s) are located
- Classification: Determine inside that localization what pre-defined class it is
So in order to build an object detector we must train our model with data that includes both images of the objects and bounding boxes (also referred to as annotations) of where that object is in the image.
In the example above, we are showing where exactly in the photo is specific classes (the count of the dots). Once we have a set of images and annotations, we’re ready to train with Turi Create (https://github.com/apple/turicreate).
But in order to have better accuracy when it comes to prediction, you need A LOT of test images. Multiply that by the 12 different classes that you need to test + various lighting conditions, rotation, colours, etc, and that equals a boat load of test data.
Fortunately for us, Turi Create provides a One Shot Object Detection Toolkit for us.
One Shot Object Detection (OSOD)
Basically if we provide the object images (the dots), the toolkit will randomly insert those objects into random backgrounds. It will also rotate and add skew to those starter images, providing more of a real life image. And Finally it will automatically add in our annotations so that we don’t have to manually create those ourselves.
Obviously this One Shot Object Detection won’t work in all cases. For 3D objects like faces it doesn’t do a good job. Also, cases where you’re looking for very specific objects (like tumours on x-rays for example), it’s better to have manual annotations.
But for our case, we know that a domino is a domino and besides perhaps some color differences, they all look the same — so using OSOD will work for us.
Generating the model
After installing Turi Create we run the following python script :
- These are the starting domino images for each count
- Loop through all the starting images and add a label for each
- Create a SFrame for all your images and labels
- Pass those start images into the OSOD create method (Note: This will take about 2 hours on a decent MacBook Pro)
- Save the model so we can test it with Turi Create later
- Export the Core ML model so we can use it in an app.
Testing
So, how did we do? A quick test on an app gave pretty good results, but not the best.
In one example, we can see it did a perfect job:
But in another example, it confused the 6 with the 10. Why? Not 100% positive, but the colours are similar,
But the model does do a good job at finding multiple dominos
The model also doesn’t do a good job at farther distances, and you need to be within 6 inches of the tile in order for it to detect it.
What would I do differently next time?
- Add more manual images — with various angles, and various color adjustments
- Make my starter images a bit more like the tile. Include the rounded corner, blank line, instead of cutting that out.
- More data images with rotation.
- Make some greyscale images, or start with black and white tiles.
Conclusion
Machine learning is just getting started in its practical form. But what’s exciting, is that anybody can participate. Apple is giving developers the tools to easily make models to share with the world.
What I predict in the future is the next iteration of “Googling” the answer — having machine learning tell you what something is exactly…and maybe a few less friendships will be broken up because of it!
