With threading I pick up 9 'hits', without threading I pick up 214. What's going on?

Question

The basic purpose of my script is to filter through a range of numbers (say, 5000),the numbers that are valid are saved to a list called hit_list. The real range I'm looping through is much bigger than 5000, so I need concurrency to make the time manageable.

I don't know the proportion of valid numbers in any given range, so when my (threaded) script returned 9 numbers to hit_list I didn't question it. However as a final check I ran the script without threads, just like a normal script. It returned 214 numbers to hit_list!

EDIT: To be clear, the problem is that numbers are not being found correctly, rather than not being stored correctly.

I've been very generously helped with the construction of this programme, both on SO,here and Reddit,here.

Below is the script with threads. I suspect the problem is something to do with locking (though I was under the impression that concurrent.futures solved this problem automatically) or maybe with the number of workers/chunks. But as I'm sure you can tell by now, I'm a beginner, so it could be anything!

import concurrent.futures as cf
import requests
from bs4 import BeautifulSoup
import time
from datetime import datetime
import xlwt

hit_list =[]
print('-List Created')
startrange= 100000000
end_range = 100005000
startTime = datetime.now()
print(datetime.now())
url = 'https://ndber.seai.ie/pass/ber/search.aspx'

#print('Working...')

def id_filter(_range):

    with requests.session() as s:
        s.headers.update({
            'user-agent': 'For more information on this data collection please contact #########'
        })

        r = s.get(url)
        time.sleep(.5)


        soup = BeautifulSoup(r.content, 'html.parser')
        viewstate    = soup.find('input', {'name': '__VIEWSTATE'          }).get('value')
        viewstategen = soup.find('input', {'name': '__VIEWSTATEGENERATOR' }).get('value')
        validation   = soup.find('input', {'name': '__EVENTVALIDATION'    }).get('value')

        for ber in _range:            


            data = {
            'ctl00$DefaultContent$BERSearch$dfSearch$txtBERNumber': ber,
            'ctl00$DefaultContent$BERSearch$dfSearch$Bottomsearch': 'Search',
            '__VIEWSTATE'                                         : viewstate,
            '__VIEWSTATEGENERATOR'                                : viewstategen,
            '__EVENTVALIDATION'                                   : validation,
        }

            y = s.post(url, data=data)

            if 'No results found' in y.text:
                #print('Invalid ID Number')
                pass
            else:
                #print('Valid ID Number')
                hit_list.append(ber)




if __name__=='__main__': #not 100% clear on what exactly this does, but that's a lesson for another day.

#Using threads to call the function    
workers = 20
with cf.ThreadPoolExecutor(max_workers=workers) as e:

    IDs = range(startrange,end_range)
    cs = 20 
    ranges = [IDs[i+1 :i+cs] for i in range(-1, len(IDs), cs)]
    results = e.map(id_filter, ranges)
#below is code for saving the data to an excel file, I've left it out for parsimony.

Answer 1

This solution is a re-written, rather than debugged version of my original code. Re-building from the ground up was both easier and more educational for me- but I can't tell you what went wrong with the original code.

The script seems to work well- it picks up the magic number of 214 hits in a loop of 5000.

Although it does answer my question, the code is far from perfect and I'm happy to accept answers that build/improve upon it.

Not being sure what the SO etiquette for asking questions within answers is, I'm calling the following optional discussion points (rather than questions!):

1. Thoughts on requests add-on: It seems to be a much more straightforward version of what I've tried to do?
2. Best way to optimise efficiency of threads? Are there any rules of thumb regarding the ratio of workers to chunks, or is it just trial & error?
3. Best way to raise exceptions while using requsts & threading?

Working (but imperfect) Solution

import concurrent.futures as cf
import requests
from bs4 import BeautifulSoup
from datetime import datetime



hit_list =[]
processed_list=[]
startrange= 100000000
end_range = 100005000

loop_size=range(startrange,end_range)
workers= 5
chunks= 20
startTime = datetime.now()



def id_filter(_range):
    url = 'https://ndber.seai.ie/pass/ber/search.aspx'
    with requests.session() as s:
        s.headers.update({
            'user-agent': 'For more information on this data collection please contact ######'
        })    
        r = s.get(url)   

        soup = BeautifulSoup(r.content, 'html.parser')
        viewstate    = soup.find('input', {'name': '__VIEWSTATE'          }).get('value')
        viewstategen = soup.find('input', {'name': '__VIEWSTATEGENERATOR' }).get('value')
        validation   = soup.find('input', {'name': '__EVENTVALIDATION'    }).get('value')

        for ber in _range:            
            data = {
                'ctl00$DefaultContent$BERSearch$dfSearch$txtBERNumber': ber,
                'ctl00$DefaultContent$BERSearch$dfSearch$Bottomsearch': 'Search',
                '__VIEWSTATE'                                         : viewstate,
                '__VIEWSTATEGENERATOR'                                : viewstategen,
                '__EVENTVALIDATION'                                   : validation,
            }        
            y = s.post(url, data=data)            
            if 'No results found' in y.text:
                processed_list.append(ber)
                #print('Invalid ID', ber)
            else:
                hit_list.append(ber)
                processed_list.append(ber)
                print('Valid ID',ber)

if __name__ == '__main__':   
    with cf.ThreadPoolExecutor(max_workers=workers) as boss:
        jobs= [loop_size[x: x + chunks] for x in range(0, len(loop_size), chunks)]        
        boss.map(id_filter, jobs)

#note below is a simplified version of what I'm doing in real-life. In reality, I print out details similar to these and then save them in a .txt file. I left out all that for parsimony's sake.
print('Details')
run_time = datetime.now() - startTime
print('Start Range',startrange)
print('End Range', end_range)
print('Run Time', run_time) 
print('Number of Hits', len(hit_list))
print('Number of processed IDs', len(processed_list)) 
print('Number of Workers', workers)
print('Job Size:',jobs)