1

I have an Access 2010 database stored on a network drive. It does the following:

  1. Pull a list of 250,000 parts (this is fine)
  2. Analyse the part numbers in VBA
  3. Save the results of the analysis to a 2D array
  4. Write the list back out to a table in the database for use by other databases

If I run this process with the database saved to my desktop, it takes about 50 seconds to complete.

If I run it with the database saved on the network drive, it takes about 300 times as long. the element that seems to taking the longest is writing to results from array to table

Sub WritePMROutput()
Status "Writing output to tblPmrParts"
Set db = CurrentDb
Dim rstOutput As DAO.Recordset
Set rstOutput = db.OpenRecordset("tblPMRParts")

db.Execute "DELETE tblPMRParts.* FROM tblPMRParts;" 'clear the table of all data before wriing new data

intPMR = 0
Do Until intPMR = intArrPMRSze 'loop until the counter reaches the size of the array
With rstOutput
    .AddNew 'new line in output table
    !PMRID = arrPMR(intPMR, 0)
    !strayChars = arrPMR(intPMR, 1)
    !partno = arrPMR(intPMR, 2)
    !extension = arrPMR(intPMR, 3)
    If Not arrPMR(intPMR, 4) = "" Then !PartPatternID = arrPMR(intPMR, 4) 'the if not function seems to be required here as was having issues with data type with "" in a number field
    If Not arrPMR(intPMR, 5) = "" Then !ExtPatternID = arrPMR(intPMR, 5)
    .Update
    intPMR = intPMR + 1
End With
Loop

LogUpdate "PDM", "tblPMRParts"

End Sub

Can anyone suggest how I can improve on the efficiency of this operation? Or am I simply limited by the speed of my network?

Interestingly, I can copy the database to my desktop, run it, compact it, and copy it back to the network in about 5-10 mins.

Thanks in advance!

EDIT:

Following the very successful implementation of the fix suggested by Kim, I thought I would report back with the results. The situation was: On local PC, took about 60 seconds to run; across network, took about 15,000 seconds to run

After implementing a simple transaction (begin at the start of the sub, commit at the end): On local PC, 35 seconds (nearly twice as fast); across network 500 seconds - (30 times faster!)

The only issue I had with implementation was that the number of record locks required threw an error, but a quick Google search yielded a simple line of code to temporarily increase the number of record locksdAO.DBEngine.SetOption dbMaxLocksPerFile, 300000 Hope this helps someone else in future!

pa1983
  • 244
  • 4
  • 11
  • 2
    You're limited by the speed of your network if it takes 300 times longer than when it does on your desktop. Code you've posted it not relevant. If you **can** copy the db to desktop then do that as far as i know this is going to be the best solution as long as the DB is not being used by multiple users at the same time as you may loose some data but that's rather obvious. –  Feb 24 '14 at 15:28
  • Any specific reason why you would use a wildcard to delete a table's content, instead of just dropping the table and re-creating a new one? – html_programmer Feb 24 '14 at 15:43
  • mehow thanks for your response - I can copy to the desktop, but I was hoping there may have been a more elegant approach, as the database is used by 5 other people in my team and there is potential for issues with data-loss or lock issues. – pa1983 Feb 24 '14 at 15:47
  • 2
    I don't agree with @mehow- code is relevant and makes it possible to suggest some other solutions. You could try to create local table (without copying your db) and next you could use simple `INSERT INTO` sql statement aiming your db on server. In my opinion it will go very quickly. – Kazimierz Jawor Feb 24 '14 at 15:48
  • Approximately how many rows are you writing back to the table using the code in your question? All 250,000? – Gord Thompson Feb 24 '14 at 15:51
  • 1
    You can use database transactions to increase performance (BeginTrans - CommitTrans). – html_programmer Feb 24 '14 at 15:54
  • @KimGysen - the table has relationships so unfortunately can't be deleted, hence the clear all and repopulate approach. Thanks for your input! – pa1983 Feb 24 '14 at 15:59
  • @KazJaw - that's a very interesting idea - I'll give it a go in the morning (on my way home now) and report back on the results. Thanks! – pa1983 Feb 24 '14 at 16:01
  • @GordThompson - I'm writing back the vast majority of the results – pa1983 Feb 24 '14 at 16:02
  • @KimGysen - I'm not familiar with database transactions - I'll do some reading (fairly new to DBs and VBA). Many thanks again for your help – pa1983 Feb 24 '14 at 16:04
  • @pa1983 I only used transactions in PHP until now, but its principle is fairly easy. You state when you want to begin your transaction, and you define when you want to commit the operations that are a part of the transaction. The benefit of transactions is to execute multiple statements at once; a rollback mechanism ensures that no changes are made in case an error occurs somewhere in between two operations. – html_programmer Feb 24 '14 at 16:09
  • Can you open a connection to the network db file in exclusive mode, update the data, and then close the exclusive connection? If you can make that work, it should be faster. – HansUp Feb 24 '14 at 22:07
  • 1
    The first question I would ask is how did the array get filled with data (and how long does it take to fill that array). You code as is should run quite fast. I HIGH recommend you turn off row locking as it will cause HUGE extra amounts of disk I/o – Albert D. Kallal Feb 25 '14 at 01:32
  • @hansup I should be able to set up exclusive access to the DB as I can set it to run at night. Thanks for the idea – pa1983 Feb 25 '14 at 08:02
  • @AlbertD.Kallal The array was filled from a table (which was pulled from a system dump on an intranet site). I thought it would be quicker to pull all the data into an array, process it within the array, then put it back all in one go. It only takes a few seconds to read all 250,000 records in. I'll have a look at turning off record locking. Thanks! – pa1983 Feb 25 '14 at 08:05

2 Answers2

2

I refer to this source as it is explained pretty well:
http://support.microsoft.com/kb/146908

Example syntax is provided here too:

Private Sub Form_Load ()
     Dim Starttime, Endtime
     Dim db As Database
     Dim t As RecordSet
     Dim i As Integer
     Dim tempName As String
     Dim temphone As String
     Set db = Workspace(0).OpenDatabase("c:\vb\BIBLIO.MDB") ' Uses a
      ' copy of BIBLIO.MDB.
     Set t = db.OpenRecordSet("Publishers", dbOpenTable)
     Starttime = Now
     'BeginTrans  ' Add this and CommitTrans (below) for greater speed.
     For i = 1 To 100
        tempName = "testname" & Str$(i) ' Make an arbitrary unique
                                        '  string.
        tempPhone = Str$(i)             ' Make arbitrary number.
        t.AddNew ' AddNew clears copy buffer to prepare for new record.
        t!PubID = 30 + i  ' Set primary key to unique value.
        t!Name = tempName  ' Set Name field to unique value.
        t!Telephone = tempPhone  ' Set Telephone field to unique value.
        t.Update   ' Write the record to disk or to transaction buffer.
     Next i
     'CommitTrans  ' Add this and BeginTrans (above) for greater speed.
     Endtime = Now
     MsgBox "Time required= " & Format(Endtime - Starttime, "hh:mm:ss")
     t.Close
     db.Close
     End
  End Sub

You can use database transactions to reduce the number of read-writes to your table.
You perform your loops and logic in memory, but only commit your recordset to the database once.

Quote from the website:

If you do not use the BeginTrans and CommitTrans statements, this program reports 17 seconds to add 100 records on a 486/66 PC. When you add BeginTrans and CommitTrans as shown in the program comments above, the program takes less than 1 second on that same computer. Performance may vary on different computers.

html_programmer
  • 18,126
  • 18
  • 85
  • 158
  • 1
    Kim, apologies for being so slow getting back to you with feedback. I was pulled away from the project by other commitments. I've just finished implementing and testing your suggestions - the results were superb! I first tested transactions with the DB on my PC - it went from ~60 secs to ~35 secs. When tested across the network, it improved from 15,000 secs to ~550! Thank-you very much for your answer! – pa1983 Jun 05 '14 at 12:36
-1

The problem with Access is that it's a Client-Side Database. Whenever Access runs a process against a table, it has to pull the entire contents of the table to your local space, perform the operation, and then push the results back out to the database. This is why most people end up opting to use SQL Server as a back-end, as it's Server-Side and can do processing (in the form of Stored Procedures and Views) on the server to minimize what you bring to the front-end.

The most efficient way to delete records in a table is using a pre-contstructed DELETE query. You may pick up a little time by doing that, as opposed to using a db.Execute statement. However, you can also try changing that statement to:

CurrentProject.Connection.Execute "DELETE * FROM tblPMRParts;",,,AdCmdText
Johnny Bones
  • 8,786
  • 7
  • 52
  • 117
  • 3
    re: " Whenever Access runs a process against a table, it has to pull the entire contents of the table to your local space, perform the operation, and then push the results back out to the database." - No, that is incorrect. -1 (Sorry.) – Gord Thompson Feb 24 '14 at 16:08
  • Do you have a link that says it's not? – Johnny Bones Feb 24 '14 at 16:14
  • 1
    Look here: http://stackoverflow.com/a/5833633/2144390 - "The database engine doesn't read the entire table..." – Gord Thompson Feb 24 '14 at 16:19
  • Look here, an Access MVP says it does: http://www.utteraccess.com/forum/lofiversion/index.php/t1978392.html – Johnny Bones Feb 24 '14 at 16:21
  • 1
    In the third paragraph of his first post in that thread Albert says "...this does not mean that JET necessarily pulls down a "whole" table when dealing with a single record from that table. In other words JET only needs to pull the records you telling it to deal with (assuming indexing can be used)." – Gord Thompson Feb 24 '14 at 16:25
  • He also later states that "The only way to eliminate this "double trip" is to have the particular operation run server side and not from the work station". In other words, if it's performing the View or Stored Procedure in SQL Server. Just like I said in my answer. – Johnny Bones Feb 24 '14 at 16:27
  • let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/48285/discussion-between-gord-thompson-and-johnny-bones) – Gord Thompson Feb 24 '14 at 16:27
  • Although the creation of the 2D array is most likely the biggest chunk, I would be curious to find out how long each of the other two pieces take. If you added a "debug.print "Start Part 1 " & Now()" at the start of each part and another at the end, you would know where to focus your attention first. How many folders deep is your DB on the server? That can make a big difference. – Wayne G. Dunn Feb 24 '14 at 17:29
  • Right, but that "double" trip of the record back is for only records that came down the network pipe. In other words if you pull a record from a table of 100,000 rows, then ONLY the one record comes down the network pipe. You then modify it, and then it goes back up the pipe. I mean even with SQL server, you have to pull down the record to allow the user to edit/modify it. Where SQL can really win is you can run code (or updates) 100% server side, but not in all cases. So the array() example code here would not likely benefit from SQL server unless the code was moved to a store proc. – Albert D. Kallal Feb 25 '14 at 01:30
  • @AlbertD.Kallal - As the author of that post, I gotta ask; if the OP is running a "Delete * From Table", is he pulling every record down from that table and deleting it? Gord and I had a sidebar and we weren't sure about that. – Johnny Bones Feb 25 '14 at 02:00
  • cc: @AlbertD.Kallal re: `DELETE` - I just tried that on a linked table named [Table1] with 1,000,000 rows in a back-end .accdb on the server. Total table size is approximately 70 MB. Started a trace with Wireshark and issued `DELETE FROM Table1`. Wireshark reported that the total traffic between my notebook and the server was 373 KB (both directions). – Gord Thompson Feb 25 '14 at 15:11
  • (I guess you weren't notified when I added that post to our chat later on yesterday.) – Gord Thompson Feb 25 '14 at 15:36
  • @GordThompson - Interesting. So, I wonder if action queries like DELETE and UPDATE only push packets up to the back end and don't actually pull anything down to the front end? Was **any** of that traffic downstream? – Johnny Bones Feb 25 '14 at 15:38
  • In fact, most of the traffic (~85%) was downstream, from the server to my notebook. The ACE engine on my machine had to read a certain amount of information from the .accdb file to figure out which data pages it actually needed to update (those essentially being the indexes on the table, and some other housekeeping information), but it really didn't need to write all that much information back. (Probably just a couple of "this index is empty" entries, and again some additional housekeeping.) – Gord Thompson Feb 25 '14 at 18:43
  • Deletes will require a round trip because you "touching" all records. As I stated the fact that SQL server can update all rows in a table or delete all rows in a table WITHOUT round trips does NOT suggest or imply that retrieving ONE record to edit means all records come down the network pipe. If we only edit one record, then only the one record comes down the pipe. To delete all reocrds – yes much more network traffic then SQL server. However not applications need to delete full tables, so your mileage is going to vary on the type of tasks you need to do or in the case of JET/ACE avoid. – Albert D. Kallal Feb 26 '14 at 03:19