Foreground moving object segmentation is a fundamental problem in many computer vision applications. As a solution for foreground segmentation, background modelling has been intensively studied over past years and many effective algorithms have been developed. However, accurate foreground segmentation is still a difficult problem. Currently, most of the algorithms work solely within the colour space, in which the segmentation performance is prone to be degraded by a multitude of challenges, such as illumination changes, shadows, automatic camera adjustments, and colour camouflage. However, the acquisition of large-scale datasets with hand-labelled ground truth is time-consuming and labour-intensive, by using these methods often hard to implement in practice. The proposed method develops the solution of this problem for the task of drivable area and road anomaly segmentation by proposing a self-supervised learning approach. The proposed method can automatically generate segmentation labels for drivable areas and road anomalies. Then, we train RGB-D data based semantic segmentation neural networks and get predicted labels. We firstly develop a pipeline named Self-Supervised Label Generator (SSLG) to automatically label drivable areas and road anomalies. Then, we use the segmentation labels generated by the SSLG to train several RGB-D data-based semantic segmentation neural networks.