Finding an exciting event in a lengthy spherical (360°) surveillance video having an unlimited Field of View (FoV) is challenging and time-consuming. Hence, this paper proposes a novel spherical video synopsis framework to condense the 360° surveillance video. It also incorporates an action recognition module, making monitoring important activities easy, followed by less critical ones. The framework provides flexibility for generating synopsis from spherical videos based on the viewer’s preferences. The preferences include customized FoVvisualization of the generated spherical synopsis video and FoV-based personalized synopsis video generation. In the first case, the viewer experience is customized by viewing the interested FoV based on the head movement overthe generated spherical synopsis video. In the second case, the viewer experience is personalized by generating a synopsis video only for the viewer’s interested FoV dynamically. Overall, the proposed framework creates acondensed immersive video using various optimization techniques by reducing collisions between objects, preserving all events with interaction, chronological ordering, and showing the viewer only the specified number of objects per frame in the synopsis video recognizing important actions. Exhaustive experimental analysis of the used optimization algorithm is performed for the proposed video synopsis framework applicable for unlimited FoV. The analysis includes a video synopsis of varying lengths and predefined lengths.