Polyphenols are promising nutritional bioactives exhibiting beneficial effect on age-related cognitive decline. This study evaluated the effect of a polyphenol-rich extract from grape and blueberry (PEGB) on memory of healthy elderly subjects (60–70 years-old). A bicentric, randomized, double-blind, placebo-controlled trial was conducted with 215 volunteers receiving 600 mg/day of PEGB (containing 258 mg flavonoids) or a placebo for 6 months. The primary outcome was the CANTAB Paired Associate Learning (PAL), a visuospatial learning and episodic memory test. Secondary outcomes included verbal episodic and recognition memory (VRM) and working memory (SSP). There was no significant effect of PEGB on the PAL on the whole cohort. Yet, PEGB supplementation improved VRM-free recall. Stratifying the cohort in quartiles based on PAL at baseline revealed a subgroup with advanced cognitive decline (decliners) who responded positively to the PEGB. In this group, PEGB consumption was also associated with a better VRM-delayed recognition. In addition to a lower polyphenol consumption, the urine metabolomic profile of decliners revealed that they excreted more metabolites. Urinary concentrations of specific flavan-3-ols metabolites were associated, at the end of the intervention, with the memory improvements. Our study demonstrates that PEGB improves age-related episodic memory decline in individuals with the highest cognitive impairments.