This is too broad and long of a question for SO.
However, the first step?
You need to write out what features you going to support. So, say if your system is going to support over time hours, then you need to design your software to handle such cases.
The general approach here is to first build your data model. That means in simple terms you need to design the database first.
Then, you have to determine is this system going to calculate the taxes? In fact, this tax calculation part can be the most challenging.
I have written from scratch 2 times a payroll system in my long software career.
I found the government documents on how such tax calculations work to be VERY complex. You have often the federal government taxes, and then you have different tax rates and calculations for the region (or state, or province) where the employees work.
Most governments do have a tax site portal in which you can get such documents for tax calculations. And in some cases, they will even have a working screen in which you can say type in some hours, and it will perform the tax calculations. Or, often they will have a set of tables. You will need these documents, since then you can take their math and formulas for tax calculation, run your test code, and see if your tax calculations match up their math and calculations.
As noted, I found this part of the project to be the most difficult part of my payroll project.
The number of formulas required was quite large, and I recall that the documents had about 8 pages in total of such math formulas.
So, good math skills tend to be required for such a project.
So, you have to decide (or know) if your payroll system is going to do the tax calculations, or it is JUST a tool to track the hours, over time hours etc., and track the pay rates (for regular hours, and overtime hours).
As I stated and pointed out, the tracking of such information is not difficult, but requires a good database model design BEFORE you write ONE line of code.
So, my top advice?
Outline the features you going to support. You really need to do this as the first step in your project.
Then with features defined that you plan to support?
Build the database model. And build this data model before you write ANY software, or even one line of code!
As noted, the data operations of a payroll system are mostly standard data operations, and are standard fair in terms of code. However, it still going to be a complex data model if you are new to this.
So, calculations of pension, holiday pay, unemployment insurance, and tax rates for a given pay was the most difficult part of these applications.
In many cases, a small business before having a computer payroll system? They take a set of tables and tax rates etc. from a government document. Turn out these government documents are created with computer programs. Thus, these "manual" tax rate tables for payroll are very valuable, since you can use these tables to check your math.
And most governments also have documents on how to calculate the tax rates with software. They don't give you or have the software, but they do outline the math and formulas.
As noted, the math for tax calculations was about 8 pages long, and it was page after page of math formulas. In other words, you better be a ace with math, since those parts of the payroll system were VERY complex math formulas, and ones that you have to turn into computer code.
There is a sample database model here you can look at to follow and get some ideas from:
https://www.vertabelo.com/blog/payroll-data-model/
Each feature can be lot of extra work, so removal and not supporting too many features goes a long way to reducing the cost and scope of such a project.
so, as a first step, you need to get those tax documents.
The small business department of your local government tax office will have such documents, and in most cases they are free and can be downloaded.
For me, trying to wade though these documents, and follow the math for the tax calculations was the most difficult part of the project.
However, if your software is not to do the payroll tax calculations, then your first goal is to design the data model. That database design is thus one of the most important parts of your project.
Either use the above example as a starting point, or look for some other data models to start with. If you are new to database modeling, then you want to gain some experience on building a relational database system, and how you relate tables to each other (this so called data modeling takes some practice to get good at).
Building a good database design is the top number one thing in such a project, and get that data model part correct, the rest of the project will like magic fall into place