One of the best sources to go to for information on this kind of question is the Catholic Encyclopedia on the New Advent site. The entry for "Ash Wednesday" reads:
The name dies cinerum (day of ashes)
which it bears in the Roman Missal is
found in the earliest existing copies
of the Gregorian Sacramentary and
probably dates from at least the
eighth century.
The existence of a penitential ceremony at the beginning of Lent seems to have been in existence as a tradition prior to 1091:
There can be no doubt that the custom
of distributing the ashes to all the
faithful arose from a devotional
imitation of the practice observed in
the case of public penitents. But this
devotional usage, the reception of a
sacramental which is full of the
symbolism of penance (cf. the cor
contritum quasi cinis of the "Dies
Irae") is of earlier date than was
formerly supposed. It is mentioned as
of general observance for both clerics
and faithful in the Synod of
Beneventum, 1091 (Mansi, XX, 739), but
nearly a hundred years earlier than
this the Anglo-Saxon homilist Ælfric
assumes that it applies to all classes
of men. "We read", he says,
in the books both in the Old Law and
in the New that the men who repented
of their sins bestrewed themselves
with ashes and clothed their bodies
with sackcloth. Now let us do this
little at the beginning of our Lent
that we strew ashes upon our heads to
signify that we ought to repent of our
sins during the Lenten fast.
And then he enforces this
recommendation by the terrible example
of a man who refused to go to church
for the ashes on Ash Wednesday and who
a few days after was accidentally
killed in a boar hunt (Ælfric, Lives
of Saints, ed. Skeat, I, 262-266). It
is possible that the notion of penance
which was suggested by the rite of Ash
Wednesday was was reinforced by the
figurative exclusion from the sacred
mysteries symbolized by the hanging of
the Lenten veil before the sanctuary.
The tradition of marking oneself with ashes as a sign of penitence is found throughout the Bible.
