Abstract: Traditional image processing methods have low efficiency in detecting surface defects of various metal sheets during the production process, making it difficult to meet the needs of industrial production. In order to improve the accuracy of metal sheet surface defect detection, a metal sheet surface defect detection method based on optimized Faster R-CNN algorithm is proposed. Using the residual network ResNet50 as the backbone feature extraction network. Firstly, the Feature Pyramid Network and Deformable ConvNets v2 are fused to improve the detection ability for small objects and irregular defects. Then, RoI Align and K-means++ clustering algorithms are used to optimize the candidate boxes and achieve precise defect localization. Finally, the proposed model is applied to multiple experiments in the NEU-DET dataset. The experimental results shows that the mAP of the optimized Faster R-CNN algorithm on this dataset is 78.7%, which is 7.7% higher than the original network, and its detection performance is better than SSD, YOLOv5s, and YOLOv7 object detection algorithms.