Firstly there are a few pre-requistes to be followed for the upgrade. So please be aware of the breaking changes between v3 and v4 as mentioned here.
You need run the pre-upgrade validator
and also confirm before the upgrade process. The steps are outlined here
You also need to upgrade your local project before you publish it to Azure.
While running it on windows you need to ensure that you have enabled .NET 6.0 which is a must for the v 4.x Function app runtime and then republish your app.
az functionapp config set --net-framework-version v6.0 -g <RESOURCE_GROUP_NAME> -n <APP_NAME>
More information, for your reference is here.