The gluon polarization in the nucleon was measured using open charm production by scattering $160\mathrm{GeV}/c$ polarized muons off longitudinally polarized protons or deuterons. The data were taken by the COMPASS Collaboration between 2002 and 2007. A detailed account is given of the analysis method that includes the application of neural networks. Several decay channels of ${\mathrm{D}}^0$ mesons are investigated. Longitudinal spin asymmetries of the D meson production cross sections are extracted in bins of ${\mathrm{D}}^0$ transverse momentum and energy. At leading order QCD accuracy, the average gluon polarization is determined as $⟨\Delta g/g{⟩}^{\mathrm{LO}}=-0.06\pm 0.21(\mathrm{stat}.)\pm 0.08(\mathrm{syst}.)$ at the scale $⟨{\mu }^2⟩\approx 13(\mathrm{GeV}/c{)}^2$ and an average gluon momentum fraction $⟨x⟩\approx 0.11$. The average gluon polarization is also obtained at next-to-leading order QCD accuracy as $⟨\Delta g/g⟩$ $\mathrm{NLO}=-0.13\pm 0.15(\mathrm{stat}.)\pm 0.15(\mathrm{syst}.)$ at the scale $⟨{\mu }^2⟩\approx 13(\mathrm{GeV}/c{)}^2$ and $⟨x⟩\approx 0.20$.

• Received 28 November 2012

DOI:https://doi.org/10.1103/PhysRevD.87.052018