Suggest you first go through the history of your release definition, if there have unexpected edits. If something is causing a change to the definition, it should show there as an update and might provide some clues as to why these tasks disappearing.
Then also check if these custom tasks are also missing in the task list and corresponding build definition.
One possibility, if your custom task is calling the TFS API where you have to specify the api-version parameter. You maybe still using the wrong version value. Updating API version on the TFS installation is only done when you upgrade the TFS version/update. When you are on TFS 2017 Update 1 so that specific version of the API may not be available in TFS 2017 Update 3.
For example if your task is using an old version of the REST api. For compatibility, elements of definitions that are not supported by a rest client's version are removed before sending the definition to the client. This unfortunately means that if the client then saves the definition the aspects that were not supported by its api version are removed from the definition.
You could also create a test new release definition then add same tasks as the old release definition to see if the issue still exists. This will narrow down the issue is related to task itself, or just some cache. For cache issue, please refresh it as Pranay suggested.
