As the continuous consumption of fossil fuels has caused serious diseases, environmental pollution, and distributing the ecological balance, renewable energy sources (RESs) such as solar, wind, hydroelectric, and geothermal energy have started to attract great attention all over the world. The use of renewable and low-carbon energy sources plays a significant role in supplying electrical energy demands for sustainable and environmentally friendly energy production. Photovoltaic (PV) power generation is one of the remarkable energy types to provide clean and sustainable energy. However, losses of electricity production are generally caused by the presence of various anomalies influencing the operation systems in PV plants. Therefore, rapid fault detection and classification of PV modules can help to increase the reliability of the PV systems and reduce operating costs. In this study, an efficient PV fault detection method is proposed to classify different types of PV module anomalies using thermographic images. The proposed method is designed as a multi-scale convolutional neural network (CNN) with three branches based on the transfer learning strategy. The convolutional branches include multi-scale kernels with levels of visual perception and utilize pre-trained knowledge of the transferred network to improve the representation capability of the network. To overcome the imbalanced class distribution of the raw dataset, the oversampling technique is performed with the offline augmentation method, and the network performance is increased. In the experiments, eleven types of PV module faults such as cracking, diode, hot spot, offline module, and other classes are utilized. The experimental results show that the proposed method gives higher classification accuracy and robustness in PV panel faults and outperforms the pre-trained deep learning methods and existing studies.