1

Let say i have 100k records in table, after fetching that records from table i am pushing it to an array with some calculations, and then send them to server for further processing.

I have test the scenario with(1k) records, its working perfectly, but worrying about if there is performance issue, because the page which do calculation and fetching records from db run after each 2 mins.

My Question is can I use array for more than 2 Millions records?

hakre
  • 193,403
  • 52
  • 435
  • 836
Pir Abdul
  • 2,274
  • 1
  • 26
  • 35
  • You can use arrays for as much memory as you have allocated to PHP. However, that's what resources are there for, not to waste memory. Use `while($row = fetch)`. – Shef Oct 03 '11 at 09:28
  • Not enough info. In general: ofcourse you can, there is no real global problem with 500k. But performance and if it works at all will depend on the machine you run it, the amount of memory it has, the amount of memory PHP is allowed to use, the type/size of the data, what should be done with the data, the speed of your database, etc etc etc. – Nanne Oct 03 '11 at 09:29
  • 2
    Array or no array, 2m records are quite a lot (depending on what kind of data is in them). Do you really *have* to store them in memory all at once? – deceze Oct 03 '11 at 09:29
  • Yes, i need to fetch from table and then do some operation on each record and store it to an array. – Pir Abdul Oct 03 '11 at 09:31
  • possible duplicate of [php array size limit](http://stackoverflow.com/questions/5925885/php-array-size-limit) and [What is the max key size for an array in PHP?](http://stackoverflow.com/questions/467149/what-is-the-max-key-size-for-an-array-in-php) – hakre Oct 03 '11 at 09:32
  • The page is scheduled and run after 2 min, so i am worrying about the performance issue, any substitute to not using arrays. – Pir Abdul Oct 03 '11 at 09:34
  • PHP Size and Make key size is some thing different , i need suggestion that should i use array for fetch 2 millions records and do calculation on that array and send them to server for further processing. – Pir Abdul Oct 03 '11 at 09:37

3 Answers3

6

There's no memory on how much data an array can hold, the limit is server memory/PHP memory limit.

Why would you push 100k records into an array? You know databases have sorting and limiting for that reason!

fire
  • 21,383
  • 17
  • 79
  • 114
  • 1
    +1 for the second part. You should not read all those records for sure, learn how to use the database instead. It is intended for querying. – kapa Oct 03 '11 at 09:31
  • +1.. as a reference, to change the memory limit "on the fly", you could use `ini_set('memory_limit', '2048M');` – pleasedontbelong Oct 03 '11 at 09:34
  • This does not answer the question – hakre Oct 03 '11 at 09:34
  • Sorting and limiting is some thing else, i do not need sorting. i need to fetch each records and store in array and send them to the server for further processing. – Pir Abdul Oct 03 '11 at 09:35
  • Then you can stagger it, do 100 at a time by changing the LIMIT in your sql query... `SELECT * FROM blah LIMIT 0,100` `SELECT * FROM blah LIMIT 100,100` `SELECT * FROM blah LIMIT 200,100` – fire Oct 03 '11 at 09:37
  • But i need to fetch all, not 100 at time, and send them to server for further processing. – Pir Abdul Oct 03 '11 at 09:41
2

Let say i have 100k records in table, after fetching that records from table i am pushing it to an array with some filters.

Filters? Can't you just write a query that implements those filters instead? A database (depending on vendor) isn't just a data store, it can do calculations and most of the time it's much quicker than transferring the data to PHP and doing the calculations there. If you have a database in, say, PostgreSQL, you can do pretty much everything you've ever wanted with plpgsql.

Berry Langerak
  • 18,561
  • 4
  • 45
  • 58
  • Still, MySQL is powerful enough to do most of the work for you. If I'm not mistaken, MySQL also supports stored procedures in which you can encapsulate the calculations; you'll only have to retrieve the results in PHP. – Berry Langerak Oct 03 '11 at 09:43
2

My Question is can I use array for more than 2 Millions records?

Yes you can, 2 Million array entries is not a limit in PHP for arrays. The array limit depends on the memory that is available to PHP.

ini_set('memory_limit', '320M');
$moreThan2Million = 2000001;
$array = range(0, $moreThan2Million);
echo count($array); #$moreThan2Million

You wrote:

The page is scheduled and run after 2 min, so I am worrying about the performance issue.

And:

But I need to fetch all, not 100 at time, and send them to server for further processing.

Performance for array operations is dependent on processing power. With a fast enough computer, you should not run into any problems. However, keep in mind that PHP is an interpreted language and therefore considerably slower than compiled binaries.

If you need to run the same script every 2 minutes but the runtime of the script is larger than two minutes, you can distribute script execution over multiple computers, so one process is not eating the CPU and memory resources of the other and can finish the work in meantime another process runs on an additional box.

Edit

Good answer, but can you write your consideration, about how much time the script will need to complete, if the there is no issue with the server processor and RAM.

That depends on the size of the array, the amount of processing each entry needs (in relation to the overall size of the array) and naturally the processor power and the amount of RAM. All these are unspecified with your question, so I can specifically say, that I would consider this unspecified. You'll need to test this on your own and building metrics for your application by profiling it.

I have 10GB RAM and More than 8 Squad processor.

For example you could do a rough metric for 1, 10, 100, 1000, 10000, 100000 and 1 million entries to see how your (unspecified) script scales on that computer.

I am sending this array to another page for further processing.

Metric as well the amount of data you send between computers and how much bandwidth you have available for inter-process communication over the wire.

hakre
  • 193,403
  • 52
  • 435
  • 836