Brain lesions detected in magnetic resonance images often vary in type and rarity across different cohorts, posing a challenge for deep learning techniques that are typically specialized in recognizing single lesion types from homogenous data. This limitation restricts their practicality in diverse clinical settings. In this study, we explore different deep-learning approaches to develop robust models handling both subject and imaging variability, while recognizing multiple lesion types. Our research focuses on segmentation and detection tasks across four distinct datasets, encompassing six cohorts of subjects with white matter hyperintensities, multiple sclerosis lesions, or stroke abnormalities. Our findings reveal that a cascade approach, comprising a fully convolutional network and a fully connected classifier, offers optimal accuracy for robust multiclass lesion segmentation and detection. Notably, our proposed model remains competitive with models trained solely on one dataset and applied to the same dataset while showing robustness against domain shifts. Additionally, in related tasks, our model consistently produces results comparable with the state-of-the-art methods. This study contributes to advancing clinically applicable deep learning techniques for brain lesion recognition, offering a promising solution for handling lesion diversity in uncontrolled clinical environments.