In this thesis, we present fundamental methods to allow automatic scene analysis in surveillance using keypoint cameras. The goal of such future cameras is to discard the original images and output a stream of keypoints instead. Since these points can be used for object categorization and recognition, scene knowledge on a higher semantic level can be obtained. As the traditional image-based foreground segmentation pipeline cannot be applied for keypoints, we develop new methods from scratch to segment keypoints into foreground and background. The goal is to replace the image-based change detection for detecting foreground objects with a keypoint-based motion segmentation. This has advantages regarding privacy issues, and it overcomes limitations from change detection regarding shadows or non-static cameras.We first tackle the problem of segmenting keypoint correspondences between two images according to common motion. Next, multiple of such frame-to-frame motion segmentations are combined such that trajectories are formed which are segmented according to their motion over time. Then, we derive an approach to describe objects which are not densely textured. Here regular keypoint detectors do not find suitable keypoint locations and leave undescribed areas. With our no-feature (NF) approach, we are able to fill these areas with an NF keypoint, which can be matched with NF keypoints of undescribed areas in further images. Finally, we show two applications: online learning of the keypoints of an object by motion, and pixelwise segmentation of an image based on motion-segmented keypoints.Our experiments show that our motion segmentation approach has an error of only 23% of comparable state-of-the-art methods, and it can be run in real-time. It is further shown that NF keypoints perform superior to regular keypoints with respect to precision and recall and that object segmentation becomes much more accurate if motion-segmented NF keypoints are added to the set of regular keypoints.