How I can clear scrapy jobs list?

Question

How I can clear scrapy jobs list? When I start any spider I have a lot jobs with specific spider and I know how can I kill all them ? After reading documentation I have done next code, which I run in a loop:

cd = os.system('curl http://localhost:6800/schedule.json -d project=default -d spider=google > kill_job.text')
file = open('kill_job.text', 'r')
a = ast.literal_eval(file.read())
kill='curl http://localhost:6800/cancel.json -d project=default -d job={}'.format(a['jobid'])
pprint(kill)

cd = os.system(kill)

but looks like that it doesn't works. How can I kill all jobs because even if I have finished manually scrapy's process in the next start all jobs come back. Find this https://github.com/DormyMo/SpiderKeeper for project management. Does anybody know how to include existing project ?

@davedwards I have started it in 5000 loop and watched how changes count of jobs and it is the same — kolas, Nov 20 '18 at 19:18
have you tried suggestions here: [how to remove jobs from lists?](http://groups.google.com/forum/#!topic/scrapy-users/g3YChcsFIGQ). Looks similar to your code: `for JOB in (curl http://localhost:6800/listjobs.json?project=myproject)->running: $ curl http://localhost:6800/cancel.json -d project=myproject -d job=JOB` — chickity china chinese chicken, Nov 20 '18 at 19:52
@davedwards have find this too, now working with it, later will report a result — kolas, Nov 20 '18 at 19:59

score 1 · Accepted Answer · answered Nov 21 '18 at 05:25

So, I do not know what is wrong with my first example, but I fixed problem with this:

cd = os.system('curl http://localhost:6800/listjobs.json?project=projectname > kill_job.text')
file = open('kill_job.text', 'r')
a = ast.literal_eval(file.read())
b = a.values()
c = b[3]
for i in c:
    kill = 'curl http://localhost:6800/cancel.json -d project=projectname -d job={}'.format(i['id'])
    os.system(kill)

score 1 · Answer 2 · answered Jan 17 '22 at 09:41

Took @kolas's script and updated it for python 3:

import json, os
PROJECT_NAME = "MY_PROJECT"

cd = os.system('curl http://localhost:6800/listjobs.json?project={} > kill_job.text'.format(PROJECT_NAME))
with open('kill_job.text', 'r') as f:
    a = json.loads(f.readlines()[0])

pending_jobs = list(a.values())[2]
for job in pending_jobs:
    job_id = job['id']
    kill = 'curl http://localhost:6800/cancel.json -d project={} -d job={}'.format(PROJECT_NAME, job_id)
    os.system(kill)

I added `os.remove("kill_job.text")` at the end too. – Jack Mar 09 '22 at 11:15 — Jack, Mar 09 '22 at 11:15

How I can clear scrapy jobs list?

2 Answers2

Linked