The article you cited, which links six coloring agents with hyperactivity, the study's result was apparently overstated by The Lancet's press release. According to Dr Paul Illing, Registered Toxicologist and Fellow of the Royal Society of Chemistry,
The Lancet's press release is overstating the case presented in the paper. The paper is, in effect, four studies - two on three-year-olds and two on eight-year-olds, in each case using two non-identical mixtures.
The paper shows some statistical associations, it is not a demonstration of cause and effect. Mixture A, but not mixture B gave a statistical association with hyperactivity, but the paper does not determine which component of the mixture is responsible or if it is an interactive effect. It supports a hypothesis that certain food additive mixtures may be associated with hyperactivity, without identifying which additives and certainly without identifying this as a feature of all food additives. The hypothesis should be further investigated.
Extrapolating from the small study population to the general public, let alone to individuals within the general public, is very difficult. The differences seen were probably very small when compared with the inter-individual variation seen within the appropriate general population.
Nowhere does this study show the food sources of the food additives that children may be exposed to. Thus, the important question of whether the diets containing these quantities of food additives are typical of a well balanced diet or are skewed by the eating habits of these two age groups cannot be addressed.
Dr Susan Jebb, Nutrition Scientist at the Medical Research Council, adds:
Such additives are most likely to be found in foods that we would like to see children eating less of i.e. soft drinks, confectionery and so on, and so it reiterates the general healthy eating messages of encouraging healthier food choices.
In the countries where they've been banned, it's usually in spite of the lack of evidence. For example, in 2008, when the Food Standard Agency is announced it was considering banning those six specific E-numbers, here's how The Guardian reported the news:
The agency also paved the way for negotiations to get the colourants removed across the European Union after its board decided precautionary action was needed, despite the lack of evidence of a biological causal mechanism of hyperactivity through the additives.
Dame Deirdre Hutton, the FSA chair, said it was the agency's duty to put consumers first. The colours were not necessary and it was therefore "sensible" to remove them from food.
As such, I wouldn't worry too much of the presence of E-numbers in the food you eat.
If there were any solid evidence of risk, it wouldn't be allowed for consumption. When there is uncertainty, some countries will decide to ban the additive; others won't. That does not mean that the product is harmful; it only means that it's possibly harmful. Adjust your dietary habits in consequence of how you feel about products whose affects on health are possibly harmful.