0

I am using pythons billiard multiprocessing package to run some queries in a postgres db. I have a list with years let's say [2020, 2021, 2022] and a list of queries that take the years in a parametrised where clause. What I do is pass the the list to Pools via map and what I was expecting was to see in my db running parallel queries for different years, but instead what I see is 16 processes running for the same years. Ex (where year = 2020) x 4 | (where year = 2021) x 4 etc... Could you please help me understand why this is happening?

Code used:

years = [2019, 2020, 2021]
params = zip(years, repeat(<table_name>), repeat(<sql>), repeat(<parameters>))
pool = Pool(processes=4)
try:
  r = pool.map(run_sql, params)
except Exception as exc:
  pool.close()
  raise Exception(f'{name} generated an exception: {exc}')
else:
logging.info(f'''Loaded {name}-{r}
                    --------------------------------
                    ''')
pool.close()

def run_sql(params):
        year, name, sql, parameters = params[0], params[1], params[2], params[3]
        try:
            sql_filtered = sql + f''' where date_part('year', created_at) = {year}'''
            <module to run sql>     
            return years
        except Exception as e:
            raise e
Saraki
  • 297
  • 1
  • 7
  • 21
  • Can you share the output that show the 16 processes running? – Ulisse Bordignon Sep 23 '22 at 07:20
  • Please add the definition of your `repeat` function. Also using `f-string`s and other string templating in SQL queries opens the door to SQL injection (including 2nd level injection). – ljmc Sep 23 '22 at 07:27
  • Thank you for your comments, I know for the sql injection but the machine is secured, also I don't have access to said machine to show the processes running the only results that I can see are from the db queries running and from logs that I print. In the logs I see each year printed once (as expected) but in DB I see for each year 4 queries running – Saraki Sep 23 '22 at 07:33
  • Secured or not, 2nd level injection happens. – ljmc Sep 23 '22 at 08:30

0 Answers0