The　combination　of　nonlinear　spectrum　and　convolutional　neural　network　(CNN)　is　efficient　for　fault　diagnosis　of　nonlinear　system.　However,　in　traditional　method,　the　nonlinear　spectrum　calculation　was　accomplished　by　identification　algorithm　outside　the　CNN,　which　reduced　the　diagnosis　efficiency.　To　solve　this　problem,　a　novel　CNN　with　the　function　of　spectrum　calculation　and　fault　diagnosis　is　designed,　in　which　the　spectrum　calculation　network　and　the　fault　diagnosis　network　are　connected　in　series.　By　extracting　the　optimized　parameters　of　network,　the　nonlinear　spectrum　based　on　generalized　frequency　response　function　(GFRF)　is　obtained　in　the　former　network.　Then,　the　GFRF　spectrum　is　automatically　put　into　the　latter　network　for　feature　extraction　and　diagnosis.　Hence,　after　determining　the　structure　of　the　CNN,　only　by　system　input　and　output,　the　fault　diagnosis　can　be　realized,　which　avoids　the　complex　process　in　traditional　method.　What＇s　more,　a　new　error　cost　function　model　is　designed　to　guide　the　network　parameters　optimization　in　the　direction　of　feature　classification,　which　is　conductive　to　improve　the　diagnosis　accuracy.　The　proposed　network　model　is　applied　to　the　heavy-duty　industrial　robot　system,　and　the　best　performance　is　demonstrated　by　several　experiments.　©　2021