Coffee is associated with decreased breast cancer risk, but the impact of body mass index (BMI) in combination with coffee consumption on prognosis is unclear. The suppressive effect of coffee constituents on the insulin-like growth factor receptor 1 (IGF1R) levels in breast cancer cells may play a role. The aim was to investigate the prognostic impact of coffee consumption and possible associations with tumor-specific IGF1R protein expression and BMI in a population-based cohort in Sweden, comprising 1,014 primary breast cancer patients without pretreatment enrolled 2002-2012 and followed for up to 13 years. Patients with higher coffee consumption had lower tumor IGF1R levels (P = 0.025), but only among the normal-weight patients (P = 0.005). Coffee did not impact the recurrence-risk overall. However, tamoxifen-treated patients with ER+ tumors drinking ≥ 2 cups of coffee/day had lower recurrence-risk [adjusted HR (HRadj) 0.57, 95% CI, 0.34-0.97] compared with patients with lower intake, although only among normal-weight patients (HRadj 0.37, 95% CI: 0.17-0.78; Pinteraction = 0.039). Similarly, coffee consumption ≥ 2 cups/day was associated with significantly lower recurrence-risk among the 640 radiotherapy-treated patients irrespective of BMI (HRadj 0.59, 95% CI 0.36-0.98) and in the 296 normal-weight patients (HRadj 0.36, 95% CI 0.17-0.76) but not in the 329 overweight or obese patients (HRadj 0.88, 95% CI 0.42-1.82) although the interaction was not significant (Pinteraction = 0.093). In conclusion, coffee consumption was negatively associated with tumor-specific IGF1R levels only among normal-weight patients. Though, IGF1R did not explain the association between coffee intake and improved prognosis among normal-weight tamoxifen- or radiotherapy-treated patients. Studies of IGF1R-targeting therapies may benefit from taking BMI and coffee consumption into account.