2

Morning All,

I've got a fairly new environment running the latest ESXi, all WS19S servers AD joined and everything seems to be working as it should and is very responsive as all of the servers are over provisioned in resources. Recently, we installed an application that runs from a file share on one of the servers, app seems to do well until it has to loop through a folder that has 40+GBs of PDFs (roughly 250K of files).

Here's the interesting part, when you access the app from any client or server as a domain user(did a lot of testing for this), it can take up to 40 seconds for the app to iterate through those PDFs (I know it's a poor design), need to mention, everything else seems fairly responsive. Now, log in as a domain admin on the same machine as the user did, it only takes a few seconds to go through the same file. Have had many discussions with the software vendor and they are as stumped as I am.

Has anyone seen this type of behavior before and if so, how do you overcome it?

I've disabled things like the AV and other security products to see if maybe they were scanning or limiting read/writes but nothing changed. Also, I only a have a few GPO items configured, mostly having to do with passwords.

Stumped is not the word right now!!!

Greg Askew
  • 35,880
  • 5
  • 54
  • 82
Davids Learnin
  • 15
  • 1
  • 1
    To confirm your assumption, try to do more thorough testing using the same account. Create a brand new test account, asign it domain admin rights, and measure the time. Then remove admin rights, relogin (important), make sure admin rights are gone by running `whoami /groups`, and try again. – J-M Feb 27 '23 at 17:46
  • 1
    Thanks but that has already been done, ran as described above after I removed the test account from the domain admin group (confirmed by using dumpsec). Also ran on a clean install in another environment with the same outcome. – Davids Learnin Feb 27 '23 at 18:49
  • What is the app actually doing with those files? Do you notice this slowdown only when using the app, or does it happen also when f.e. browsing the network share? Also, what happens if you access the folder on the server itself instead of using the network? – Massimo Feb 27 '23 at 21:10
  • 1
    Have you checked if assigning local admin to the account make things run faster? Please update your question with test results to shine more light on this. Anyway, my guess is this not related to some network or storage throttling, but the app is doing something else, which it can't do fast enough when run as a non-admin user. This action (or attempted action) is appear to be slowing the app. For further debuging I would go with Process Monitor from Sysinternals. It might point you to some failed actions. Especially, if you compare app activity running with and without admin rights – J-M Feb 27 '23 at 22:27
  • The app is creating a ticket, the folder in question, stores image files of the ticket in pdf format, what the process is doing, its itinerating through the folder to see if a name of a pdf file exist. If it doesnt find the file, it creates a new one. There are no errors being generated any where, I can open one of their log files and see the timestamps of each process, seeing 40-60 seconds on searching the folder for domain users vs 2-3 seconds for the same process for a domain admin. I should also add, the users can read ever file, open, modify, create and delete as needed in that folder. – Davids Learnin Feb 27 '23 at 22:47
  • Forgot to add, the app has to be ran as an local administrator, does change how it acts at all. – Davids Learnin Feb 27 '23 at 22:53
  • Iterating through a folder to see if a file name is in use? This is not poor design, this is *astonishingly bad* design. It looks like checking for the file existing, or even trying to open the file and checking for an error, was just too much of a leap... – Massimo Feb 27 '23 at 22:57
  • Can you run the app on the server itself (so to rule out networking issues)? – Massimo Feb 27 '23 at 23:00
  • 1
    Also, you can try using https://learn.microsoft.com/en-us/sysinternals/downloads/procmon to check what is going on at the file system level; maybe the standard user is hitting access denied errors which cause the slowdown. – Massimo Feb 27 '23 at 23:04
  • I find it odd that there isn't any information in the question about the security of the files in question, and if the Domain Admins group has elevated/different permissions on said documents. Domain Admins should not have permissions, that is an inferior design. – Greg Askew Feb 28 '23 at 01:03
  • Massimo, yes, we've tried running on the server in question with the same result, could also eliminate network issues because the way it runs with domain admins. Greg, domain admins have full rights in our organization as well as most, not inferior but essential in smaller environments (lack of personal which dictates segregation of duties) – Davids Learnin Feb 28 '23 at 02:10

2 Answers2

1

We had the same issue as described. For us turning off access based enumeration on the file share fixed the issue.

https://theitbros.com/config-access-based-enumeration-windows-server-2016/

JamesPorter
  • 11
  • 3
0

I'll venture a guess: there are different ACLs on the individual files.

If this is the case, when you are a standard user you can only access some of them; instead when you have admin rights you get the full load.

Depending on what the application is doing with them, having to process 1k or 10k files instead of 250k could make a significant difference.

Massimo
  • 70,200
  • 57
  • 200
  • 323
  • Thanks for your feedback but that is kind of opposite of the problem I feel meaning if it was restricted to a user, their time would be shorter, also, if they weren't seeing all of the files, the application would create a new imagine because it wouldn't exist because they don't have access Good thought but I dont think it applies.. – Davids Learnin Feb 27 '23 at 22:33
  • My bad, I completely misread your question and thought the app was slower for admins than users... – Massimo Feb 27 '23 at 22:52