Clinical evidence points to the premise that caffeine may benefit cognition, but whether these findings extend to real life settings and amidst factors that impact caffeine metabolism is unclear. The aim of this study was to investigate the impact of recent caffeine drinking on cognitive ability while additionally accounting for lifestyle and genetic factors that impact caffeine metabolism. We included up to 434,900 UK Biobank participants aged 37-73 years, recruited in 2006-2010, who provided biological samples and completed touchscreen questionnaires regarding sociodemographic factors, medical history, lifestyle, and diet. Recent caffeine drinking (yes/no in the last hour) was recorded during a physical assessment. Participants completed at least one of four self-administered cognitive function tests using the touchscreen system: prospective memory (PM), pairs matching (Pairs), fluid intelligence (FI), and reaction time (RT). Multivariable regressions were used to examine the association between recent caffeine drinking and cognition test scores. We also tested interactions between recent caffeine drinking and a genetic caffeine-metabolism score (CMS) on cognitive function. Among white participants, recent caffeine drinking was associated with higher performance on RT but lower performance on FI, Pairs, and PM (p ≤ 0.004). Similar directions of associations for FI (p = 0.09), Pairs (p = 0.03), and PM (p = 0.34) were observed among non-white participants. No significant and consistent effect modification by age, sex, smoking, test time, habitual caffeine intake, or CMS was observed. Caffeine consumed shortly before tasks requiring shorter reaction times may improve task performance. Potential impairments in memory and reasoning tasks with recent caffeine drinking warrant further study.