Low high-density lipoprotein cholesterol (HDL-C) is a major risk factor for cardiovascular diseases (CVDs), the leading cause of global mortality. We aimed to determine the effect of coffee drinking and sex and their interaction, as well as rs1800588 and rs1800775 polymorphisms on HDL-C levels in Taiwanese adults. Data of 4262 men and 4813 women, aged 30-70 years, were retrieved from Taiwan Biobank. The interaction between sex and coffee drinking on HDL-C was significant (p = 0.0452). Coffee consumption was significantly associated with higher HDL-C levels in only women (β = 0.81679; p = 0.0246). However, rs1800588 and rs1800775 variants were significantly associated with HDL-C in both sexes. In women, β-values were 0.99080; p = 0.0059 and 3.16277; p < 0.0001 for rs1800588 CT and TT genotypes, respectively and -1.80954; p < 0.0001 and -2.81512; p < 0.0001 for rs1800775 AC and CC genotypes, respectively. In men, β-values were 1.32430; p < 0.0001 and 3.24976; p < 0.0001 for rs1800775 CT and TT genotypes, respectively and -1.96232; p < 0.0001 and -2.71536, p < 0.0001 for the AC and CC genotypes, respectively. In conclusion, coffee drinking was significantly associated with higher high-density lipoprotein (HDL) levels in women but not men after adjusting for confounders including rs1800588 (LIPC) and rs1800775 (CETP) variants.