The water deer (Hydropotes inermis) is one of the rarest species of deer in the family Cervidae. Only two subspecies exist in East Asia, and few studies have examined the genetic characteristics of the species. Here, we investigated the genetic diversity, phylogeny and population differentiation of the Korean subspecies (H. inermis argyropus). Seventeen mitochondrial D-loop haplotypes (822 bp) were detected and analyzed from 107 individual samples, together with a Chinese subspecies (H. inermis inermis) haplotype. The genetic diversity of the Korean subspecies is lower (π = 0.756%, h = 0.867) than that of the Chinese subspecies estimated in a previous study. This low genetic diversity may result from historical anthropogenic disturbances and/or a founder effect during the glacial period. The phylogenetic tree and median-joining network showed no location-specific distribution of D-loop haplotypes, but revealed two major lineages, A and B, of water deer. The A and B lineages were separated from each other at the beginning of the Pleistocene era (2.1–1.3 million years ago), with a genetic divergence of 1.332 ± 0.340%. The genetic divergence within lineages A and B was 0.525 ± 0.167% and 0.264 ± 0.113%, respectively. This suggests that climate change affected the division of the two lineages. Water deer sampled from the three Korean regions (26 locations) were slightly distinct in their genetic structure (AMOVA: F_ST = 0.28416, P < 0.00001; Φ_ST = 0.19239, P < 0.00001). Such slight population differentiation may be derived from differential dispersal ability in males and females. The use of genetic markers, such as nuclear microsatellite and Y-linked DNA markers, and samples collected from various localities in East Asia should improve our understanding of the water deer’s genetic characteristics.