0

I need to fetch around 250K numbers of records continuously from an API. from the client side i click on a button,using ajax server fetch the records from the api and stored as files in the server's file system. but because of this huge numbers of records. the system get stopped execution after 50K. is there is any method in php to fetch data parallel processing or using multiple threads.

Akash
  • 1
  • 1
  • Fetching a huge amount of records in one go is not a good idea, do you really need all that data in one go? Can you cache some of it (it may be a little out of date but does that matter)? – Nigel Ren Jul 01 '19 at 06:57
  • @Nigel Ren, Thanks. I already split the job into 6 and use cookie to store the information and resume this again using cookies. but if multi-threading is possible i can fetch multiple records at a time. – Akash Jul 01 '19 at 07:12
  • I'm not sure if you answered my comment - it is more about the need to fetch all the data in one go and not the method. – Nigel Ren Jul 01 '19 at 07:15
  • Yes. i really need to fetch all the data in one go. – Akash Jul 01 '19 at 07:20

2 Answers2

0

You can check/use following approaches to check which suits your requirement better: 1. Use async ajax calls 2. Increase MAX_EXECUTION_TIME and MEMORY_LIMIT directive in php so that it can process the complete request. 3. Increase upload data size.

KnightIn
  • 18
  • 2
  • i make ajax call only at once after that server select an id from database and fetch its corresponding record through api call(looping this around 250K times). i put sufficient MAX_EXECUTION_TIME. but server stopped before that. is there is any method like multi-threading. so that i can fetch 2 or 3 records at a time. – Akash Jul 01 '19 at 06:42
  • see, it could be better if you limit your data fetched in database and whatever data you want at a time call this api in a time interval – CodeBreaker Jul 01 '19 at 06:45
  • Thanks. is there is any multi threading methods in PHP to solve this. I found that pthreads only available in php-cli. – Akash Jul 01 '19 at 06:54
0

The problem is not with fetching the data from database. It is on other side, i.e., writing the data on file.

File do not save too much data at once. you have to make it in chunks.

Murad Faridi Shah
  • 11
  • 1