im newbie with multi-threads and im trying to make a program that use it. Technically, it has a class with executor and a worker class. The worker class takes the info to work on, process it, if there are more infos to processes the worker calls the executor again and the program keeps going until the text is finished, after that the workers decrease the AtomicInteger check variable and and if equals zero change the status attribute to DONE and saves it. The executor class receive the attributes needed to the worker class, check if the text/site was proceeded before, if not increments an atomic integer attribute and put the new text/site reference into a Vector for further checks again.
The Executor class:
public class DoScrapper implements IDoScrapper {
private static Logger log = LoggerFactory.getLogger(DoScrapper.class);
private final ThreadPoolExecutor executor = (ThreadPoolExecutor) Executors.newFixedThreadPool(30);
void doScrapper(final ScrapperWorker scrapperWorker) {
executor.execute(scrapperWorker);
}
@Override
public void sendToProccess(
ScrapperEntity entity,
String urlToProccess,
String rootUrl,
Vector<String> urlVisited,
IRequestService requestService,
IDoScrapper doScrapper,
Database database) {
urlVisited.add(urlToProccess);
int qtd = entity.addUrlInAction();;
log.info("sending to procces url {} for keyword {} , with {} sites opened to visited",
urlToProccess, entity.getKeyword(), qtd);
ScrapperWorker newWorker = new ScrapperWorker(
entity, urlToProccess, rootUrl,
urlVisited, requestService,
doScrapper, database
);
doScrapper(newWorker);
}
}
The worker class:
public class ScrapperWorker implements Runnable{
private static final Logger log = LoggerFactory.getLogger(ScrapperWorker.class);
final ScrapperEntity entity;
final String url;
final String rootUrl;
final Vector<String> urlVisited;
private final IDoScrapper doScrapper;
private final Database database;
public ScrapperWorker(ScrapperEntity entity,
String url,
String rootUrl,
Vector<String> urlVisited,
IRequestService requestService,
IDoScrapper doScrapper,
Database database) {
this.entity = entity;
this.url = url;
this.rootUrl = rootUrl;
this.urlVisited = urlVisited;
this.requestService = requestService;
this.doScrapper = doScrapper;
this.database = database;
}
@Override
public void run() {
log.info("thred doing scrapper for url {} keyword {} ",
url, entity.getKeyword());
....
do actions and can or cannot call based on the check() method doScrapper();
...
decreaseUrlInActionAndVerifyStatus();
}
private void decreaseUrlInActionAndVerifyStatus() {
int sitesToCheck = entity.decreaseUrlInAction();
log.info("decreasing visitedSites on entity {}, sites to check {}, sites visited {} ",
entity.getId(), sitesToCheck, urlVisited.size());
if(sitesToCheck == 0 ){
log.info("last url, setting STATUS as done ... sitess visited: {} ", urlVisited.size());
entity.setStatus(StatusScrapper.DONE);
database.updateData(entity);
}
}
private boolean check( String urlComplete, Vector<String> urlVisited) {
return urlComplete != null
&& isNotVisitedUrl(urlComplete, urlVisited);
}
private boolean isNotVisitedUrl(String url, Vector<String> vector) {
return ! vector.contains(url);
}
}
The methods on entity class to decrease and increase:
private volatile AtomicInteger countUrlInAction;
public int addUrlInAction(){
return countUrlInAction.addAndGet(1);
}
public int decreaseUrlInAction() {
return countUrlInAction.decrementAndGet();
}
The problem im having its that sometimes, principally when I put more Threads on the thread pool of the executor or more parallel processing [ more entities to be processed in parallel ] the AtomicInteger become random, sometimes stopping in 1, others in 7 and so on, and the status never change to DONE. Isn't for AtomicInteger to increase and decrease correctly in this case or Im missing something? If I cant use AtomicInteger for this, which way is better to do this check?
There aren't any error in the console log, so all threads must be processing all the texts and decreasing the AtomicInteger variable
