A Comparative Analysis of Object Identification Labelling Platforms: Basketball Perspective

Introduction

Image labelling is the process of adding meaningful information to images and it mainly involves data labelling by recognizing and labelling individual components. It helps the machine to “see” by identifying and labelling specific details in the image. Manual labelling is a common way to label images. It refers to the process of adding meaningful information to images by hand through labels. It defines how AI “sees”, but any inaccuracy can lead to misinterpretations. But manual labelling can pose severe challenges. Unintentional and imperfect human labelling can hinder the model’s accuracy which is also time consuming and requires expertise. The largest available image dataset, ImageNet would have needed an estimated 19 years to be labelled by a single person. Data pre-processing itself consumes over 80% of the project duration [1]. This underscores the importance of exploring optimised labelling platforms for object identification. They aim to significantly reduce labelling time and also support users to create more accurate and consistent labels. Even minor improvements in speed and quality can lead to significant gains in overall efficiency.

The study aims to achieve a measurable performance boost in basketball detection accuracy by choosing the optimal platform for data annotation through a comparative analysis of various data labelling platforms available. This can drastically pave way for increased transparency in VAR-assisted decisions, contributing to a more just and reliable game.

Literature Survey

According to Reference, despite the remarkable capability of human vision to interpret the complexities of a scene with a single glance, traditional computer vision algorithms still face challenges in achieving this ability. This can reduce the overall efficiency and accuracy during the detection of various objects. To address this challenge, various deep learning techniques have been explored by researchers, with promising results achieved using the YOLOv8 algorithm-a latest iteration of the You Look Only Once(YOLO) family of object detection algorithms which offers remarkable speed, accuracy and flexibility [2].

The study addresses the importance of deep learning , various algorithms and tools in the sports field. Traditionally, sports experts like coaches, managers, sports health professionals etc. have relied on conventional analytical methods to analyse data and make critical decisions. But this can be an arduous and time-consuming process. Recognizing this challenge, machine learning emerges as a powerful and promising solution that can revolutionise the sports industry through automation [3].

Reference highlights a crucial challenge in Machine Learning and Deep Learning: requirement of a large amount of data for establishing a foundation for reliable learning platforms. This process involves meticulous data annotation, where elements within the data are identified, localised, and tagged with labels. These labels become the “ground truth” for the model. The model uses it to check its predictions for accuracy and any errors in this process may lead to a poor quality dataset and this can ultimately hinder the performance of the predictive models it is used for [4].

A related work delves deeper into the selection process for data labelling tools. There are some other factors that should be taken into consideration other than the tool’s core functionalities during the data labelling process [5]. Licence compliance plays a key role, ensuring the tool satisfies with our organisation’s requirements as some might be restricted to non-profit use. To accommodate diverse users, including those with less technical experience, user interface (UX) becomes another crucial factor. Data security is also equally important. We need to prioritise tools that avoid storing data on supplier servers. Costeffectiveness and flexibility of the annotation tools are the other important factors.

In study, data is recognised as the “new oil” of the 21st century because of its growing demand. This has accelerated the need to optimise the data labelling process. The goal is to generate high quality datasets faster and more efficiently, minimising time, effort, and cost. Numerous tools are out there to address this need. But our aim is to identify the most efficient and revolutionary tool that can respond to Computer Vision or Machine Learning problems. In this study we are comparing different data labelling tools and finding the best among them to implement in the project [6].

Test Case

Data collection is a very crucial and the foremost step for building our model. In this study, we focused on constructing a robust basketball dataset for image analysis. For training and development, we utilised a readily available Kaggle basketball dataset. Additionally, we captured images of real-world basketball matches to further diversify the training data. A total of 1000 images have been collected from these sources. Figure 1 shows some of the sample images.

Then comes the labelling process. We create a dedicated “basketball” class for the model to recognize and label the basketball in every snap. With this labelled dataset in hand, we move towards the training process, feeding information to our model. The model now analyses the labelled examples and it gradually understands the visual characteristics of a basketball. Finally to determine the effectiveness of the training process, we put the model to test. We present it with unseen basketball images to check whether our model identifies the basketball or not. The result of this assessment would reveal the true extent of the model’s capacities. Figure 2 shows the images after labelling.

(a)                                                        (b)                                                        (c)

Figure 1: Sample Images Taken During Data Collection

                                                 (a)                                                                     (b)                                                            (c)

Area of Interests (AOIs)

Even the slightest drop in the quality of labelled annotations can affect the model's performance [7]. So choosing the right image labelling platform can be crucial for the project's success. To identify the optimal image labelling software for our study, ai, SentiSight.ai, Labelbox and SuperAnnotate. Beyond cost analysis, we evaluated each platform's efficiency, user- friendliness, and compatibility with our workflow. Some of the highlights of the reviewed platforms are given below.

• Roboflow - A user-friendly and free platform, perfect for beginners as well as researchers. It is completely free and ideal for exploring and learning AI. It also has powerful augmentation features which adds more versatility to the datasets.

• Makesense.ai - Utilises artificial intelligence to address the challenges of large datasets and labelling tasks. It is suitable for complex projects. It is also a collaborative platform for transparent and reproducible search.

• SentiSight.ai - Used for research involving sensitive healthcare data. Their dedicated focus on medical image annotation, position them as a trusted partner for researchers in healthcare- related fields.

• Labelbox - speeds up labelling with auto-tools and shortcuts. It ensures quality through benchmarking and collaboration. It is also very flexible as it can handle any data type.

• SuperAnnotate - can handle images and video with ease. Pre- labelling, autolabelling and customisable workflows within this platform can save time and effort, boosting efficiency.

Applications

money spent on human labour for wages and infrastructure. Various labelling platforms support users in image labelling and they have several other objectives [8].

(1) labelling a large set of images within a tight timeframe.

(2) supporting the creation of a complete and balanced dataset.

(3) increasing the labelling efficiency which in turn can reduce humanworkload.

(4) ensuring accurate and reliable labels while avoiding noisy data.

(5) transforming annotations to well-organised data. Labelling platforms like Roboflow, Makesense.ai, SentiSight. ai, Labelbox and SuperAnnotate, offer many such features and objectives to the user. Their ability to identify and classify objects finds applications in numerous fields:

Autonomous Vehicles: Real-time object recognition ensures safe navigation and collision avoidance.

Retail: Identifying products on shelves helps in management of inventory and personalised shopping experience.

Manufacturing: Quality control systems detect defects in production lines through object identification.

Security and Surveillance: Video analysis can be done which improves the security.

Healthcare: Medical imaging uses object detection to identify what disease the patient has.

Comparative Study

This study presents a comparative analysis of five data labelling platforms to identify the best one that satisfies all our project requirements. The key features and limitations

 Figure 3: This Table Shows the Comparison of Various Labelling Platforms Based on the Study Conducted on February,

Conclusion

Our study compared several labelling platforms like Roboflow, Makesense.ai, SentiSight.ai, Labelbox and SuperAnnotate for basketball detection. SuperAnnotate excels in providing high- quality labelling services and integrates seamlessly with multiple platforms, while Labelbox offers robust features for complex labelling workflows and collaboration. But they might require technical expertise for better understanding of the concepts. Also, only paid version is available. If you are looking for an easy-to-use, beginner friendly platform with limited budget, then Roboflow is a good option to consider, especially for projects with smaller datasets. However it’s important to be aware of the dataset limit of 10,000 images on the free plan and 1,000 images after the starter trial.

Likewise, every platform have their own strengths and weaknesses and the ”best” platform ultimately depends on the project goals and the individual needs. In this project, we have chosen Roboflow for object identification labelling as it was more effective and appropriate during the basketball detection. Free trials are already available on platforms like Roboflow, SentiSight.ai, SuperAnnotate, Makesense.ai and Labelbox. Making use of these free trials is a recommended practise for users to explore different platforms before making a choice.

References

1. Schmelzer, R. (2019). Data Engineering, Preparation, and Labeling for AI 2019 CGR-DE100.
2. Jiang, P., Ergu, D., Liu, F., Cai, Y., & Ma, B. (2022).A Review of Yolo algorithm developments. Procedia Computer Science, 199, 1066-1073.
3. Zhao, Z., Chai, W., Hao, S., Hu, W., Wang, G., Cao, S., ...& Wang, G. (2023). A survey of deep learning in sports applications: Perception, comprehension, and decision. arXiv preprint arXiv:2307.03353.
4. Kshetri, N. (2021). Data labeling for the artificial intelligence industry: economic impacts in developing countries. IT Professional, 23(2), 96-99.
5. Dutta, A., & Zisserman, A. (2019, October). The VIA annotation software for images, audio and video. In Proceedings of the 27th ACM international conference on multimedia (pp. 2276-2279).
6. Reis, Pedro Miguel Lima de Sousa. "Data Labeling tools for Computer Vision: a Review." (2022).
7. Pande, B., Padamwar, K., Bhattacharya, S., Roshan, S., & Bhamare, M. (2022, May). A review of image annotation tools for object detection. In 2022 International Conference on Applied Artificial Intelligence and Computing (ICAAIC) (pp. 976-982). IEEE.
8. Sager, Christoph, Christian Janiesch, and Patrick Zschech. ”A survey of image labelling for computer vision applications.” Journal of Business Analytics 4.2 (2021): 91-110.