A purely syntactic difference, that despite being superficial may actually be the most common use case, is that <<< has lower precedence than .:
infixr 9 Control.Category..
infixr 1 Control.Category.<<<
This is similar to the difference between plain function application f x (which binds tighter than any infix, so it's basically infixl 10) and use of the $ operator like f $ x, which has lowest precedence infixr 0 $. That means, you can opt for which one requires less parentheses in an expression. <<< is handy when you want to compose functions that are themselves defined by some infix expression; this often happens when working with lenses.
More theoretically interesting is that the Category version works not only with functions, but also with morphisms from other well, categories. A simple example is the category of coercions: if you have e.g. a list of newtype-wrapped values and you want to get at the underlying representations, it would be inefficient to map over the list – this would create a copy of the entire list, which however contains the exact same runtime information. A coercion allows you to use the original list all the time, but without bypassing the type system – the compiler will at each point keep track in which “view” of the list the elements have what type. Coercions aren't actually functions – they are always no-ops at runtime – but they can be composed just like functions (e.g. coerce from Product Int to Int, then coerce from Int to Sum Int).
For other examples, Haskellers will usually refer to Kleisli categories. These contain functions of the form a -> m b, where m is a monad. Although you cannot directly compose e.g. readFile :: FilePath -> IO String with firstFileInDirectory :: FilePath -> IO FilePath, because there's a mismatch between FilePath and IO FilePath, you can Kleisli compose them:
import Control.Monad
main = writeFile "firstfileContents.txt" <=< readFile <=< firstFileInDirectory
$ "src-directory/"
and the same thing can be written
import Control.Arrow
main = runKleisli ( Kleisli (writeFile "firstfileContents.txt")
<<< Kleisli readFile
<<< Kleisli firstFileInDirectory
) $ "src-directory/"
What's the point? Well, it allows you to abstract over different categories and thus have code that will work both with pure functions and with IO functions. But frankly, I think Kleisli does a bad job at motivating the use of other categories: anything you can write with Kleisli arrows is typically more readable when written with standard monadic do notation, or else just with the =<< or <=< operators. That still allows you to abstract over computations that may be pure or impure, simply by choosing different monads (IO, ST or simply Identity).
There are apparently some specialist parsers that are Arrows but can't be written as monads, but they haven't really caught on – it seems the advantages do not balance the less intuitive style.
In maths, there are many more interesting categories, but unfortunately these tend not to be expressible as Categoryes because not every Haskell type can be an object. An example I like to give is the category of linear mappings, whose objects are only Haskell types that represent vector spaces, such as Double or (Double, Double) or InfiniteSequence Double. The linear mappings are essentially matrices, but with not only type-checked domain- and codomain dimension but also the option to represent different spaces with particular meaning, e.g. preventing adding a position vector to a gravity-field vector. And because the vectors don't need to be literally represented by arrays of numbers, you can have optimised representations for every application, e.g. for compressed image data on which you want to do machine learning.
Linear mappings are not a Category instance, but they are an instance of constrained category.
