Any action done by Apache as response to any user interaction will result in decreased performance for other users even if it is so small - it is almost invisible.
Both the software (memory wise) and hardware (physical limitations and capacity wise) will perform several actions to properly answer and/or log user interaction.
In this case when your users (30,000-40,000) visited the site, Apache still needed to log all of these errors in order to fulfill the logging preferences set by configuration. As the log file is just a text file on the gard disk, the hard disk was too busy to perform reads and writes on the actual site content.
The load of the server would be even higher, if Apache would trow 404 error as a document. In any case, the best way to improve server performance is to get rid of any 404 errors completely.
You can also increase the MaxRequestsPerChild parameter in Apache config file to something around 1000 to decrease the stress from one central Apache process, to multiple subprocesses. This will allow multiple Apache processes to span new ones at the same time increasing the stress to all CPU cores not only one.
With so many visitors I would also recommend changing the MPM of your Apache config to something like this :
<IfModule prefork.c>
StartServers 2
MinSpareServers 2
MaxSpareServers 5
ServerLimit 100
MaxClients 100
MaxRequestsPerChild 4000
</IfModule>