Unsigned Int uses quite often 32 bits. That means, the biggest representable number is 4294967296. This is then max_unsigned in your code.
Your first calculation X = ((A*X)+B)%max_unsigned
is therefore 1103624256*1+11254%4294967296=1103635510, which is the result, you are seeing.
In your second calculation the expression A*X
is then too big to fit into 32 bits.
That means, your code is illegal, because you do an illegal calculation. This is not covered by the C++ language standard. For example see here: https://en.cppreference.com/w/cpp/language/operator_arithmetic :
When signed integer arithmetic operation overflows (the result does not fit in the result type), the behavior is undefined.
That really means, that anything can happen. The program could behave on Mondays as you expect and on Tuesdays not. The program may crash. Your computer may crash. Your result may be incorrect. The behaviour is just not part of the language specification for C. And therefore the answer to your question must end here. That is the reason why for two years there is not even one answer to the question.
It may be the case that on certain platforms and certain compilers it is possible to predict the behaviour. And you may want to look at the Assembler code to see what happens. But all this is then outside the scope of the C language and therefore not of general interest. C is not a specification language for creating Assembler code. C has rules that must be followed. Otherwise you are not in the scope of C.
It would also not be very professional to speculate more, because in a professional environment you are normally paid to follow rules. And that includes the rules of the programming language.
If you are really interested to see what happens, you have to show the Assembler code and make it a question about Assembler programming. But there is no guarantee that your compiler always produces the same Assembler code for such illegal things. The compiler only guarantees you a predictable result if you follow the rules of the programming language and have no overflows.