0

I'm making a script that iterates through all chromosomes of a fasta file and splitting it into pieces of 10 bp, the function is called chrdata and i am saving these fragments into a single file. This fragmentation can occur on each chromosome individually completely separate for the other chromosomes, as such i'm trying threads. chrdata(faidx_t *seq_ref ,int chr_no,FILE *fp) My goal is wish to make this process faster. To achieve this i have tried multi-threading with the std::thread function.

I have tried different things.

  1. First i tried to create a thread for the first chromosome and then thread.join() then the next thread for next chromosome and so on.
  2. Then i tried to create multiple threads at once, like explained in Simultaneous Threads in C++ using <thread> This is the example below.

However as far as I understand and that I can read, I always need to use join otherwise I'll end up with "terminate called without an active exception". The issue is there is no time execution difference between example (1) and (2).

Based on my understanding its becuase despite of creating a vector with thread object they still have to join and thus wait for all the threads to execute. This means this would be concurrent execution and not parallele.

So my question is: Would anyone be able to give me suggestions to the function below where i might change to make the execution faster by using parallele execution?

Or is my understanding of join and concurrent wrong in this instance? I'm not completely sure why we cannot just skip the whole join part, if all the threads are done, why cant we just use detach()?

void function(const char* fastafile,FILE *fp,int thread_no) {
  std::vector<std::thread> threads;
  //extracting the chromosome file
  faidx_t *seq_ref = NULL;
  seq_ref  = fai_load(fastafile);
  assert(seq_ref!=NULL);
  
  int chr_total = 10; //just the first 10 chromosomes
  int chr_idx = 0;
  int chr_no = 0;

  while(chr_idx < chr_total){
    for (chr_no; chr_no < std::min(chr_idx+thread_no,chr_total);chr_no++){
      threads.push_back(std::thread(chrdata,seq_ref,chr_no,fp));
    }
    for (auto &th : threads) { th.join(); }
    threads.clear();
    chr_idx = chr_idx + thread_no;
  }
}

I havent attacked main() or chrdata() to make the code and question more clear.

pastebin.com/iY6u9CbH

RAHenriksen
  • 143
  • 2
  • 12
  • i have, and i have used the different examples online. but that doesn't help in my instance where i want to optimize the execution time. So in this specific instance i got unsure if what i am trying is compltely messed up – RAHenriksen Mar 13 '21 at 15:36
  • `th.join` blocks until the thread returns. If thats what you want the code is fine, though I dont understand why you create a couple of threads, join them and then again spwan a couple more – 463035818_is_not_an_ai Mar 13 '21 at 15:44
  • it more for some of the files containing the data there can be a lot of chromosomes, 25 for humans and if i only want to use lets say 10 threads, then im creating the first 10 (chromosome 1-10) then the next 10 and so on. – RAHenriksen Mar 13 '21 at 15:48
  • @rustyx hope this makes it more clear https://pastebin.com/iY6u9CbH – RAHenriksen Mar 13 '21 at 15:55
  • So your `chrdata()` function locks a mutex for almost the entire duration. No wonder you see no speedup, as mutex ensures only one of them is running at any given time. Furthermore, the function writes a file. Writing a file will not become faster in multiple threads, in fact writing sequentially is the fastest way. (You might want to [edit] the pastebin into the question) – rustyx Mar 13 '21 at 16:00
  • okay thats for your quick reply. Initially i tried without mutex, but then i got an execution fail due to issues with the index file for the fasta file containing all the chromosomes. But thanks for letting me know regarding writing a file. – RAHenriksen Mar 13 '21 at 16:04

0 Answers0