Training semantic segmentation models on multiple datasets has recently gained attention, driven by the need for robust and versatile models that can perform well across diverse visual domains. However, incompatible labelling policies between established datasets represent a major challenge which hinders principled training. In this project, we address this issue by automating the discovery of visual-semantic relations across datasets and constructing a universal taxonomy that consists of classes that describe standalone visual concepts within a dataset collection. Our approach allows us to construct models which produce predictions based on the recovered universal taxonomy and can be trained in a weakly supervised manner.