I have an azure data factory pipeline that reads a 100 mb CSV from blob storage and inserts it into an Azure SQL DB that has tons of space (200 GB). I also have a 1 MB version of the same files, just with most of the data removed, as a test version.
When I run the pipeline with the full file, I get the error: DF-Executor-InternalServerError. It gives me no other useful information.
However, when I run the short version, it runs just fine. Along with a few other discussions on this error, I think the problem is simply that I need to increase execution/computer resources and/or timeout time. I'm just not sure where I should be doing that in the chain.
Has anyone run into anything like this?
