A tennis vision model tracks shot speed and body position from iPhone video

Why it matters

Agent-connected vision tools are shrinking the work required to train specialized models from personal video. The remaining challenge is converting reliable tracking into useful, validated advice.

Alan, a Montreal-based builder who publishes as AA (@measure_plan), built a computer vision pipeline that turns iPhone footage of a tennis practice session into measurements of ball speed, stroke type, shot placement and body position at contact.

https://x.com/measure_plan/status/2097715692069859769

AA demonstrated the project on September 9th in a 33-post thread on X. The input was ordinary phone video rather than footage from instrumented courts or dedicated cameras. The resulting clips show labels and tracking overlays following the player, racket and ball through each shot.

The project extends a long run of visual experiments from AA, whose GitHub profile lists dozens of repositories spanning gesture-controlled 3D interfaces, animation tools, games and MediaPipe projects. That background matters here because the tennis tracker is a working personal experiment rather than a startup launch or commercial product. AA has published the pipeline and its limitations as he develops it in public.

From phone clips to a fine-tuned detector

AA said he uploaded several videos from one practice session into a Roboflow workspace. Roboflow's Astra model automatically labeled frames for four classes: player, ball, wall and racket. He then fine-tuned RF-DETR, Roboflow's open source family of detection transformers, and used a Python script to add the tracking graphics and statistics after inference.

Roboflow describes RF-DETR as a model family for object detection, segmentation and keypoint detection. Its Apache 2.0 license permits commercial use and modification, while its pre-trained checkpoints can be fine-tuned on custom datasets. For AA's project, that custom dataset consisted of footage captured from his own practice rather than a large, prepared tennis dataset.

The toolchain also explains why the prototype came together faster than AA's previous sports vision projects. He used Roboflow's agent integration with Claude Code to control labeling and fine-tuning from the terminal. Roboflow's MCP integration exposes workspace operations to compatible coding agents, including dataset creation, uploads, training jobs, model evaluation and workflow assembly.

Astra removes another manual step. Roboflow's documentation for the auto-labeling model says users can specify object classes in text and receive proposed bounding boxes without first training a detector or drawing example boxes. Roboflow also warns that those labels require review because the model can return duplicates, loose boxes and missed objects.

AA's workflow is a compact example of what that automation changes for individual developers. A builder can collect footage, propose labels and start fine-tuning a specialized detector without spending the first stage of the project drawing every box by hand or wiring each platform operation into a separate script.

The model sees the swing. Coaching comes later

The current system remains an offline analysis tool. AA said processing a video and adding annotations takes several minutes, and he has not optimized the workflow. The overlays and statistics are added after the practice session rather than delivered as live feedback.

The training data is also narrow. AA used a few clips of his own session from one viewpoint and said he plans to add professional footage and other camera angles. Those additions would test whether the detector can keep tracking reliably when lighting, player appearance, court geometry and camera position change.

Body mechanics present a harder problem. AA used MediaPipe for pose estimation, but detailed analysis of foot and joint positions during the backswing and contact point may require two cameras and a 3D reconstruction. A single view can hide depth and occlusion, especially when the racket, arm and torso overlap at the moment of contact.

The project currently measures what happened. AA plans to add an ideal strike zone, compare each swing against it and generate specific coaching advice. He also floated AirPods for spoken feedback and an augmented-reality layer that could place practice targets on a wall, both of which would require lower latency than the present post-processing pipeline.

AA said reviewing the footage already exposed weaknesses in his footwork and backhand. That is useful feedback, though it came from watching the annotated video rather than an automated coaching judgment. The next version will need to translate detections and pose landmarks into recommendations that remain valid across different players and camera setups.

For now, the prototype's strongest result is the compressed build process. A few phone videos, automated labels, a fine-tuned detector and a Python analysis layer produced a personalized sports tracker. Turning those measurements into a coach will require a broader dataset, faster inference and evidence that its advice improves the swing rather than simply describing it.