TL;DR:
- Age-related cognitive decline begins early in adulthood and before age 60 in healthy educated adults. However these changes vary from individual from individual and is also dependent on various other factors such as childhood IQ, disease, genes, inflammation and diet/lifestyle.
Within the range defined by ‘normal cognitive ageing’—i.e. in people who would not meet the criteria for dementia or any of the varieties of ‘mild cognitive impairment’—people differ greatly in the degree to which their brains decline with age. Identifying the risk factors for, and mechanisms of, individual differences in age-associated cognitive decline is among the greatest challenges to improve the wellbeing of older people. Source: Age-associated cognitive decline
- Age-associated decline is found to be less in cognitive functions such as verbal ability, some specific numerical abilities and general knowledge but other cognitive functioning such as certain aspects of memory, executive functions, processing speed and reasoning tends to decline from middle age.
There is little age-associated decline in some mental functions—such as verbal ability, some numerical abilities and general knowledge—but other mental capabilities decline from middle age onwards, or even earlier. The latter include aspects of memory, executive functions, processing speed and reasoning. All of these so-called ‘fluid’ mental abilities are important for carrying out everyday activities, living independently and leading a fulfilling life. When one fluid mental domain declines others tend to do so also. Second, slowed speed of information processing appears to account for a substantial proportion of age-associated decline in all affected cognitive domains, and the slowing has begun by the 30s, as has the age-associated decline in some other aspects of cognitive function. Source: Age-associated cognitive decline
To answer your question, yes there is certainly a change to the brain structure when one ages.
To understand what happens on the outside, it's important to know what happens on the inside. The brain's volume peaks in the early 20s and gradually declines for the rest of life. In the 40s, when many people start to notice subtle changes in their ability to remember new names or do more than one thing at a time, the cortex starts to shrink. Other key areas also show modest changes. Neurons (nerve cells) can shrink or atrophy, and there's a large reduction in the extensiveness of connections among neurons (dendritic loss). The normally aging brain has lower blood flow and gets less efficient at recruiting different areas into operations. Source: Memory Changes in Older Adults
Research shows that age-related decline accelerates at older ages.
However, what does appear clear is that several different types of results converge on the conclusion that age-related cognitive decline begins relatively early in adulthood, and certainly before age 60 in healthy educated adults. Source: When does age-related cognitive decline begin?
However, these changes vary from individual to individual and cannot be generalized among all.
Using neuroimaging and increasingly sensitive psychological tests, researchers have refuted the model that people, as they get older, go into a general mental decline. Instead, psychologists are developing a model of specific deficits that show very different rates of decline and also vary widely among individuals. They also suspect that middle-aged sensitivities about memory loss may be exacerbated by comparisons with one's youthful performance. It may be more realistic to compare one's performance to healthy age-matched peers instead. Source: Memory Changes in Older Adults
Also with regards to your claim that "I look at the President for example, and he is 55 and he appears just as sharp as people in their 20s. And then my dad is 74 and is just as sharp with his working & long-term memory, and with learning new things - I can't notice that there is anything different with their cognitive abilities", its not surprising since recent research shows that cognitive performance is better in those born more recently.
Research comparing people of the same chronological age in different historical timeframes — for example, 50-year-olds born in 1890 versus 50-year-olds born in 1940 — reveal that cognitive performance is better in those born more recently. Thus, people with the same chronological age may have different cognitive ages, depending at least in part on the historical framework they were born and lived in. The findings hold true even for smaller differences time differences. A recent study of 90 year-olds — one group born in 1905 and another in 1915 — revealed a significant increase in cognitive performance for the group born just 10 years later.
Much of Dr. Lindenberger’s research involves exploring the mechanisms that help people maintain high levels of cognition. Given the studies Dr. Staudinger referred to and his own work showing “massive improvement” in cognitive test results in people who were 75 in 2010-13 versus those who were 75 in 1990-93, Dr. Lindenberger quipped, “You might say that 75 is the new 55, based on the data and with just a slight exaggeration.” Source: How does our cognitive ability compare with that of past generations?
Studies also show that age is not a major factor for cognitive function decline.
Many studies have shown, however, that on the whole, our cognitive abilities do not show any major decline until we are in our 50s or early 60s. And even then, in healthy people, performance diminishes very slowly. For example, in one study, at age 81, only 30 to 40% of the subjects showed any significant decline in their cognitive abilities. In other words, two-thirds of the participants in the study had only minimal decreases in their cognitive faculties—or rather, in some of their cognitive faculties, because several remained relatively intact while others even improved with age. Source: The Effects of Normal Aging on Our Cognitive Abilities
Incidental and working memory tends to be affected by age and implicit while semantic memory appears to be less affected by the aging process.
One of the forms of memory that is most affected is incidental memory, the kind that lets you remember things automatically and with practically no effort. This is the form of memory that you use, for example, to describe the scenes from a film in detail as you are coming out of the movie theatre. Incidental memory peaks during the teen years or early 20s and then decreases. As people age, it therefore becomes harder for them to recall numerous details unless they make a conscious effort to do so.
Working memory—the kind that lets us remember a telephone number for a few seconds, or follow the thread of a conversation—also is often affected by age. The slowdown in processing speed that results from aging of the brain seems to make information disappear from working memory before we have even been able to consolidate it in long-term memory. The decline in the performance of working memory might therefore also at least partly explain why long-term memory also declines with age.
Our ability to retrieve information from memory is also sometimes affected, which explains the phenomenon of having a word “at the tip of your tongue”. On the other hand, implicit memory—the kind involved in conditioning and in motor learning—appears to be less affected by aging, as is our semantic memory of knowledge that we use frequently in the course of our lives. As for our vocabulary, it continues to increase throughout our lives. Source: The Effects of Normal Aging on Our Cognitive Abilities
To answer the later specific claim noted by user Oddthinking, research shows that crystallized intelligence which is the accumulation of facts and knowledge peaks later in life. The analysis by age is explained by Christian Jarrett as, "Analysing performance by age, the researchers discovered that the ability to identify other people’s emotions peaks between the ages of 40 to 60 (i.e. there is a long plateau in peak ability); vocabulary ability peaks in the late 60s / early 70s; performance at digit symbol coding (seen as a basic measure of mental agility) peaked at 18; visual working memory peaked at 25; and working memory for numbers peaked in the early to mid-30s."
Recent evidence shows that whereas short-term memory for names and inverted faces peaks around the age of 22 years, neither short-term memory for faces nor quantity discrimination peaks until around the age of 30, a fact difficult to assimilate into the fluid-/crystalized-intelligence dichotomy. Source: When Does Cognitive Functioning Peak?
The Asynchronous Rise and Fall of Different Cognitive Abilities Across the Life Span
Also there is no defined age in which humans are performing at peak on all cognitive tasks.
Our findings in Experiment 2 are consistent with the possibility that people
born in 1945 have unusually large vocabularies, people born in 1980 have unusually good working memory, and people born in 1990 have unusually fast processing speed. Our findings have practical and theoretical implications.
On the practical side, not only is there no age at which humans are performing at peak on all cognitive tasks, there may not be an age at which humans perform
at peak on most cognitive tasks. Source: When Does Cognitive Functioning Peak?
The Asynchronous Rise and Fall of Different Cognitive Abilities Across the Life Span
To summarize, human aging is variable and influenced by external factors.
Population studies suggest that more people are maintaining higher levels of function now than ever before, and that trend will continue into the future. At the same time, human aging is “variable across and malleable within individuals,” said Lindenberger, and the brain’s ability to adapt and rise to challenges — through whatever mechanisms — can help stem cognitive decline. Source: How does our cognitive ability compare with that of past generations?.
