AI Powered Video Assistant Referee with Amazon Rekognition Custom Labels
Ever since the introduction of VARs (Video Assistant Referees) to football, many have expressed their doubts towards the objectives the initiative was supposed to fulfill. When introduced at the last FIFA World Cup in Moscow 2018, a number of football greats including Diego Maradona and Javier Zanetti expressed the improvements the technology would bring to the game they love. According to the FIFA website,
“Technology brings transparency and quality and it provides a positive outcome for teams who decide to attack and take risks.” — Diego Maradona Former Argentine professional footballer and manager.
“I support the use of VAR. I believe it is an element of greater justice for the game and for the teams.” — Javier Zanetti Inter Milan Legend and Record national player of Argentina
These comments were made way back in 2018 at the Russia 2018 World Cup. However, ever since the introduction of VAR to English Premier League, EPL, there has been an increasing distrust from fans and sport pundits in the initiative. This is largely due to inconsistent outcomes in with respect to VAR decisions. These outcomes range from penalty decisions, red/yellow card rules, to mention a few.
A recent game between Manchester United and Tottenham Hotspurs which ended 6-1 in favour Tottenham Hotspurs, saw Anthony Martial dismissed for violent conduct against Erik Lamela. In defense of Martial, he was provoked by a shove to his face by Lamela which the referee did not see happen. Lamela was then awarded a yellow card after 2 to 5 minutes of consultation with VAR. In an ideal game, both players should have been sent off.
The example above is only one out of many. The question everyone has been asking is: What if VAR could be more consistent? I would argue that this very difficult to achieve as VAR officials are humans and can attend one game at a time. This variation in VAR personnel, coupled with the fact that VARs have different experience levels is the underlying reason for the inconsistent outcomes in decisions. But wait — can AI help VARs achieve more consistent outcomes by augmenting decisions? Probably, yes. At its core, VARs make decisions based on video data (continuous stream of image frames) in real-time. These decisions are informed by their past experiences as referees or trainings. Sounds like a task for machine learning, right? Enter my experience.
Earlier in February, 2020, I to my first step towards validating my hypothesis: Could I decompose VAR into computer vision and then training a computer vision model to predict potential football fouls with minimal accuracy? As with any hypothesis, data is needed to validate. But as I started, I realized I had limited access to labeled data and I was really not enthused about writing any machine learning code at this time. Then I heard of Amazon Rekognition Custom Labels with its AutoML capabilities, taking care of machine learning for me for computer vision tasks and appeared as a perfect fit.
I collected and labeled 21 images of football fouls, just enough to validate my hypothesis. I labeled the downloaded images into 5 classes, namely: foul, no-foul, potential-yellow-card, shirt-pull, and sliding-tackle. Without writing any line of machine learning code, I trained my first model for $1, and the training metrics amazed me:
- Average Precision: 1.000 — This is the fraction of correct predictions (true positives) over all model predictions (true and false positives).
- F-Score: 1.000 —This is an aggregate measure that takes into account both precision and recall over all labels (for example, F1 score, average precision).
- Overall Recall: 1.000 —This is the fraction of your test set labels that were predicted correctly.
The training metrics above indicate that nearly 100% of the time, the model is able to identify football fouls and non-foul situations. All these achieved with zero machine learning code at $1 training cost. To non-technical people, these metrics do not mean anything unless they see sample results:
What’s More?
This project started out an experiment which I was able to validate with little data, the scale of Amazon Web Services and the ease-of-use of Amazon Rekognition Custom Labels. After seeing the potential and possibilities, I am working on opening it up as a mobile app, available to football fans round the world on EPL match days.
The goal is not to replace Video Assistant Referees who earn a living through this avenue, rather augment their work as they support match officials with more consistent outcomes from their decisions with AI. Secondly, think of the scale of this technology and how it can be made available cheaply to other football leagues across the world, delivering consistent match officiating at no extra costs.
