Content Based Surgical Video Retrieval using Scene and Motion Feature Embeddings

Abstract

Effective clinical care increasingly relies on the ability to identify relevant details within extensive heterogeneous medical datasets, a task that becomes particularly challenging with complex media formats. Traditional keyword-based retrieval methods have limitations, and content-based retrieval techniques have gained popularity. Surgical video retrieval, a novel and largely unexplored research area, offers significant utility, especially in real-time clinical scenarios. As surgical procedures are predominantly recorded via video, these recordings are often stored on servers for subsequent access, necessitating efficient retrieval of specific footage segments by surgeons. In this work, we explore various models to extract scene and motion feature embeddings, accounting for motion intensity in videos. Accurate retrieval of similar videos or images requires precise embeddings of objects within the frames. For experimental validation, we utilized the public Cholec80 dataset, which includes cholecystectomy surgery videos. The combined scene and motion features are hashed to generate frame-level binary codes, facilitating rapid search and retrieval. © 2024 IEEE.