Abstract: Taking the grinding of internal helical gears by CNC machine tools as the research object, the machining error and compensation problems in the grinding process are studied. Firstly, through the principle of homogeneous coordinate transformation, the homogeneous coordinate transformation matrix with errors between the workpiece and the tool is derived, and the geometric error model in the machining process is obtained. Secondly, the on-line monitoring of machine tool machining error is realized by using the built-in signal synchronous acquisition method, and the main error sources are compensated by the software compensation principle. Finally, through the gear detector, the tooth surface before and after error compensation is detected, and the tooth surface error after compensation is reduced. This experiment provides data theoretical support for improving the machining accuracy of machine tools.