In today’s article, we decided to reveal one of the recipes from our kitchen. We’ll talk about where we get data for futbalnet.sk.
First of all, it’s necessary to understand that the futbalnet.sk portal has already undergone a third technological change of a significant scope.
ISSF – Slovak Football Information System.
It’s an extensive system used by the Slovak Football Association (SFZ) and individual sub-associations and clubs. All competitions of all football associations in Slovakia are directly managed through this system. From the draw to the evaluations in tables and statistics.
The original portal futbalnet.sk, its first version, was directly connected to the ISSF. It was a simple portal with a straightforward overview of the results of individual competitions. However, it didn’t meet the requirements for a modern web application and its output was very limited.
That was when the SFZ established its cooperation with bart.sk, where the subject of the contract was a modern portal as the official results service for all Slovak football leagues.
We had to start, as they say, from scratch. The condition was to visualize the data from the ISSF. First of all, we needed to set out responsibilities. Where the ISSF responsibility ends and where the responsibility for the portal begins. A web service is best for such setting in communication. Any web service. We were lucky to have the opportunity to design it according to ourselves. ISSF provided basic data in a simple REST API. However, we needed to combine and process it in some way and make comprehensive reports for the future portal. So we’ve made another layer over the ISSF web service that connects the raw data from the ISSF and prepares complex data structures for the portal. We still use this extension. We also gave it a name. It’s name is Shovel. Because it transforms data from one heap to another, adding something or taking something away.
This Shovel of ours has become the basis for the modern portal. The Shovel doesn’t own any data and doesn’t have its own database. It just caches the data to Redis.
This system has brought both advantages and disadvantages. The advantage was the unambiguous and accurate data directly from the ISSF. The disadvantages came gradually, when masses of visitors started visiting the portal on Sundays after the matches. We were fine-tuning Shovel’s performance every weekend. We were adding caching layers and various optimizations. This brought about an undesirable side effect, which was a delay in the data. It sometimes took up to 30 minutes for the results to show up in the public results service.
Cooperation with SME and the new version of the sportnet.sme.sk portal have brought additional requirements, especially to improve this delay.
As you already know from our blogs and newsletters, the SportNet.online platform has Competitions as one of its applications. It’s an application through which sports associations and clubs organize competitions. Not excluding football. It has a more sophisticated API and integrates other SportNet platform services, such as CRM, ticketing, feeds and the like. Therefore, when creating the sportnet.sme.sk portal, we decided to use this API for the Slovak football results service.
But how to get the data from the ISSF into the Competitions application?
We used our favorite redis, specifically redis streams. We’ve already written about them.
The functionality is described in the schematic picture.
After the change, the ISSF publishes messages on the redis stream, informing us about what has changed in the ISSF. For example, a change of person with registration number 1315371. Or change of the match with id 567460. Another service on the server, we call it ISSF Listener, captures this message and contacts another mini service we call ISSF Connector. This is a TCP server that gets a simple command, such as update match id 567460. The ISSF Connector then reads all the necessary data from the already mentioned Shovel. We have updated it and the ISSF Connector accesses uncached data in real time. The data is processed and imported into the database for the Competitions service. Simple, right?
Of course, this didn’t do without proper tuning, both possible error detection and performance tuning. Scalability is ensured by the fact that there can be multiple ISSF Listeners that represent the consumer for the redis stream. Currently, there are 16 of them. This way, we process ISSF football update reports in 16 parallel processes. Adding another listener is just a matter of launching a new Podman container with a modified ENV configuration.
How do we handle errors?
Everything can and does go wrong. For example, a temporary unavailability of the ISSF, or erroneous data that the ISSF Connector didn’t anticipate. If any processing error occurs in the ISSF Listener, the message is stored in the so-called Error Stream for further, repeated processing. We process the error stream once every 10 minutes. If the message fails to process again, it remains in the error stream until it’s processed manually, or until processing goes through if it was a mistake caused by a temporary unavailability of some services. Of course we have reporting and monitoring about it :).
In this way, we managed to fine-tune the system of updating Slovak football data so that it’s easily scalable, so that the portal and competitions aren’t dependent on the ISSF system. To make the data consistent, fast and essentially real time.
And we already have the next-step. By connecting the sportnet.sme.sk portal directly to the Competitions module, we are able to fully expand the applications. I believe that in the next season, we’ll be able to complete and launch a new mini application – data collector. We would like to bring live results and a record of football matches to the portal in real time directly to the screens of your mobile phones.
Did you know that the Competitions application from the SportNet platform was also used as a driving force for the results service of the 2021 World Hockey Championship or the 2021 Tour De France?
