Colonoscopy is an important screening tool for colorectal cancer. During a colonoscopic procedure, a tiny video camera at the tip of the endoscope generates a video signal of the internal mucosa of the colon. The video data are displayed on a monitor for real-time analysis by the endoscopist. We call videos captured from colonoscopic procedures colonoscopy videos. Because these videos possess unique characteristics, new types of semantic units and parsing techniques are required. In this paper, we introduce a new analysis approach that includes (a) a new definition of semantic unit - scene (a segment of visual and audio data that correspond to an endoscopic segment of the colon); (b) a novel scene segmentation algorithm using audio and visual analysis to recognize scene boundaries. We design a prototype system to implement the proposed approach. This system also provides the tools for video/image browsing. The tools enable the users to quickly locate and browse scenes of interest. Experiments on real colonoscopy videos show the effectiveness of our algorithms. The proposed techniques and software are useful (1) for post-procedure reviews, (2) for developing an effective content-based retrieval system for colonoscopy videos to facilitate endoscopic research and education, and (3) for development of a systematic approach to assess endoscopists' procedural skills.