You appear to be interested in financial mathematics and algorithmic trading. Both of these fields are massive. Financial mathematics alone has a history over 100 years in length.
This is an incredibly open ended question, and you don't really have an indication of where to start. I suggest starting with the wiki pages on the relevant topics:
In a nutshell, financial mathematics deals with determining the expected future value of a stock or derivative via a formula. Algorithmic trading uses formulas from financial mathematics to suggest or automatically make trades.
The Black-Scholes model is widely used and is a good starting point for understanding the math behind the algorithms used in algorithmic trading. Start with it, and then move on to other models that you feel are better suited for what you want to accomplish.