0

I have a parser with ~50 threads. After a few hours of parserwork a have an error

sqlalchemy.exc.InternalError: (psycopg2.errors.InternalError_) Canceling query because of high VMEM usage. Used: 14MB, available 195MB, red zone: 1800MB (runaway_cleaner.c:202)

Full traceback -- https://pastebin.com/7yf7r850

I am using dataset 1.5.0 library to work with database. https://dataset.readthedocs.io/en/latest/

I create db and table functions in the init function for each thread

def __init__(self, config_data, person_id, parsing_mode):
    self.db = dataset.connect(configs.DB_CONNECTION)
    self.table = self.db.get_table('social_network_user')

then in cycle i perform insert or update operation

self.table.upsert(dict(
                        lom_id=user_id, 
                        social_network_id=follower_id,
                        social_network_login=follower_id,
                        follower_url=follower_url, 
                        follower_name=follower_name,
                        social_network_type_id=0,
                        social_uuid=f"{follower_id}.0",
                        default_total=int(self.total_followers_number),
                        subscribers_visible=subscribers_visible,
                        subscribed_ids=[int(self.person_id),]
                    ), ["lom_id", "social_uuid"])

How can I fix the error? The server is powerfull, can I increase some system limits for database?

psql --version
psql (PostgreSQL) 12.8 (Ubuntu 12.8-0ubuntu0.20.04.1)
Alex
  • 562
  • 1
  • 6
  • 25
  • 1
    Similar https://stackoverflow.com/questions/61747639/greenplum-error-canceling-query-because-of-high-vmem-usage, https://stackoverflow.com/questions/38384861/greenplum-gp-vmem-protect-limit-configuration, https://stackoverflow.com/questions/46062533/query-failing-with-error-canceling-query-because-of-high-vmem-usage , https://stackoverflow.com/questions/47151986/gpdbout-of-memory-at-segment – snakecharmerb Dec 14 '21 at 18:16
  • I didn't find solution how to fix my problem – Alex Dec 14 '21 at 18:26
  • Also it's strange for me that Used: 14MB, available 195MB. 14 < 195 but error – Alex Dec 14 '21 at 18:40
  • This is not a PostgreSQL error message. You should state and tag the database you are actually using (greenplum, I assume?). Otherwise you are unlikely to attract the attention of the people who can help you. – jjanes Dec 14 '21 at 20:20

0 Answers0