Twitter, multiple processes and database

Question

I am a beginner writing a small twitter tool for scheduled tweets and automatic retweets in python/flask.

I got stuck with issues of processes running in the background.

I want scheduled tweets and retweets to work simultaneously in the background for a given user.

I want to be able to terminate these background processes running retweets/scheduled tweets separately from each other.

How would you change the code below to achieve this?

If you look at the code below now, it works, but user can not run scheduled tweets and retweets simultaneously. Also if user decides to terminate one of the processes, let us say retweets the other process terminates as well (scheduled tweets) and vice versa.

I thought about putting the identification data for a given process into a database and recalling this identification data from the database when there is a need to terminate it, instead of using cookies session, but I do not know how to implement this idea in code.

import ........

mysql = MySQL()
app = Flask(__name__)
app.secret_key = 'xxx'

app.config['MYSQL_DATABASE_USER'] = 'xxx'
app.config['MYSQL_DATABASE_PASSWORD'] = 'xxx'
app.config['MYSQL_DATABASE_DB'] = 'xxx'
app.config['MYSQL_DATABASE_HOST'] = '0.0.0.0'
mysql.init_app(app)

@app.route('/showSignin')
def showSignin():
   if session.get('user'):
       return redirect('/userHome')
   else:
       return render_template('signin.html')

@app.route('/showscheduletweets')

def showscheduletweets():

     if session.get('user'):
      return render_template('scheduletweets.html')
    else:
       return render_template('signin.html')

     @app.route('/validateLogin',methods=['POST'])
def validateLogin():
   try:
    _username = request.form['inputEmail']
    _password = request.form['inputPassword']

    # connect to mysql

    con = mysql.connect()
    cursor = con.cursor()
    cursor.callproc('sp_validateLogin',(_username,))
    data = cursor.fetchall()

    if len(data) > 0:
        if check_password_hash(str(data[0][3]),_password):
            session['user'] = data[0][0]
            consumerkey = data [0][4]
            consumersecret = data [0][5]
            accesstoken = data [0][6]
            tokensecret = data [0][7]
            twitter = Twython(consumerkey, consumersecret, accesstoken, tokensecret)
            twitter.update_status(status="xxx says hello.")
            return render_template('userHome.html')
        else:
            return render_template('error.html',error = 'Wrong Email address or Password.')
    else:
        return render_template('error.html',error = 'Wrong Email address or Password.')

except Exception as e:
    return render_template('error.html',error = str(e))
finally:
    cursor.close()
    con.close()

#schedule tweets

@app.route('/scheduletweets',methods=['POST'])

def scheduletweets():
    if session.get('user'):
    _username = request.form['inputEmail']
    con = mysql.connect()
    cursor = con.cursor()
    cursor.callproc('sp_GetTwitter', (_username,))
    data = cursor.fetchall()

    session['user'] = data[0][0]
    consumerkey = data [0][4]
    consumersecret = data [0][5]
    accesstoken = data [0][6]
    tokensecret = data [0][7]
    twitter = Twython(consumerkey, consumersecret, accesstoken, tokensecret)

    tweet1 = request.form['inputTweet1']
    tweet2 = request.form['inputTweet2']
    tweet3 = request.form['inputTweet3']
    tweet4 = request.form['inputTweet4']
    tweet5 = request.form['inputTweet5']
    tweet6 = request.form['inputTweet6']

    Hash1 = request.form['inputHash1']
    Hash2 = request.form['inputHash2']
    Hash3 = request.form['inputHash3']
    Hash4 = request.form['inputHash4']

    fruits = [Hash1, Hash2, Hash3, Hash4]



    list = [tweet1, tweet2, tweet3, tweet4, tweet5, tweet6]
    def workit():

     while True:
        try:
            if len(list) > 0:
                z = random.randint(1, len(fruits))
                a = random.sample(fruits, z)


                b=" ".join(str(x) for x in a)
                toTweet = list[random.randint(0,len(list))-1] + " " + b

                twitter.update_status(status=toTweet)
                time.sleep(10)


            else:
                twitter.update_status(status="Oh dear... I'm afraid I'm rather empty =(")
                break
        except TwythonError as e:
            print (e)


    if 'work_process' not in session:
     process = Process(target=workit)
     process.start()
     pid = process.pid
     parent_pid = psutil.Process(process.pid).parent().pid
     session['work_process'] = (parent_pid, pid)
    return redirect('/showscheduletweets')
     #retweets
     @app.route('/retweet',methods=['POST'])
def retweet():
   if session.get('user'):

    _username = request.form['inputEmail']
    con = mysql.connect()
    cursor = con.cursor()
    cursor.callproc('sp_GetTwitter', (_username,))
    data = cursor.fetchall()

    session['user'] = data[0][0]
    consumerkey = data [0][4]
    consumersecret = data [0][5]
    accesstoken = data [0][6]
    tokensecret = data [0][7]


    Retweet1 = request.form['inputRetweet1']
    Retweet2 = request.form['inputRetweet2']
    Retweet3 = request.form['inputRetweet3']
    Retweet4 = request.form['inputRetweet4']
    Exclude1 = request.form['inputExclude1']
    Exclude2 = request.form['inputExclude2']




    def work():
     twitter = Twython(consumerkey, consumersecret, accesstoken, tokensecret)
     naughty_words = [Exclude1, Exclude2]
     good_words = [Retweet1, Retweet2, Retweet3, Retweet4]
     filter = " OR ".join(good_words)
     blacklist = " -".join(naughty_words)
     keywords = filter +" -"+ blacklist
     print(keywords)
     while True:
        search_results = twitter.search(q=keywords, count=10)
        try:
            for tweet in search_results["statuses"]:
                try:
                    twitter.retweet(id = tweet["id_str"])
                    time.sleep(60)
                except TwythonError as e:
                                            print (e)
        except TwythonError as e:
                                    print (e)

    if 'work_process' not in session:
     process = Process(target=work)
     process.start()
     pid = process.pid
     parent_pid = psutil.Process(process.pid).parent().pid
     session['work_process'] = (parent_pid, pid)
    return redirect('/showretweet')

       #terminating scheduled tweets and retweets
      @app.route('/stoptweet', methods=['POST'])
  def stoptweet():
    if 'work_process' in session:
    parent_pid, pid = session['work_process']
    try:
        process = psutil.Process(pid)
        if process.parent().pid == parent_pid:
            process.terminate()
    except psutil.NoSuchProcess:
        pass
    session.pop('work_process')
    return render_template('index.html')
else:
    return render_template('index.html')

 if __name__ == '__main__':
  app.run(host=os.getenv('IP', '0.0.0.0'),port=int(os.getenv('PORT', xxx)))

score 0 · Answer 1 · answered Dec 05 '16 at 13:00

You might want to use celery python module, and move schedule tweet and retweet as background works.

For further info, see doc: http://flask.pocoo.org/docs/0.11/patterns/celery/

You will decorate those functions related to celery, rather than flask.

As example:

In your script:

import my_schedule_module

and then in my_schedule_module.py:

from celery import Celery, Task
from celery.result import AsyncResult

from celery.task.base import periodic_task

import sqlite3 # Here I use sqlite, can be sql
import redis # Here I am using redis, you can use another db as well > check documentation

from datetime import timedelta # used to schedule your background jobs, see in configuration below


app_schedule = Celery('my_schedule_module')


'''
Celery Configuration
'''

# a mockup configuration of your background jobs, as example use retweet each 60s
app_schedule.conf.update(
    CELERY_ACCEPT_CONTENT = ['application/json'],
    CELERY_TASK_SERIALIZER='json',
    # CELERY_ACCEPT_CONTENT=['json'],  # Ignore other content
    CELERY_RESULT_SERIALIZER='json',
    # CELERY_TIMEZONE='Europe/Oslo',
    # CELERY_ENABLE_UTC=True,
    CELERYD_TASK_TIME_LIMIT = 600,
    CELERYD_TASK_SOFT_TIME_LIMIT = 600,
    CELERYD_MAX_TASKS_PER_CHILD = 1000,
    CELERYD_OPTS="--time-limit=600 --concurrency=4",
    BROKER_URL = 'redis://localhost:6379/0',
    CELERY_RESULT_BACKEND = 'redis://localhost',
    CELERYBEAT_SCHEDULE = {
        'add-every-60-seconds': {
        'task': 'my_schedule_module.retweet',
        'schedule': timedelta(seconds=60)
        },
    }
)

@app_schedule.task()
def retweet(tweet):
     # your tweet function

@app_schedule.task()
def scheduletweets():
     # your background job
     # pseudo code
     tweets = get_tweets()
     process_tweet_list = []
     for tweet in tweets:
          process_tweet_list.append( retweet.s(tweet) ) 
     job = group(process_tweet_list)  #group is celery.group, see documentation
     result = job.apply_async() # process job list async
     print 'result', result.ready(), result.successful()

You can also use callback functions - as example, you might want to update datetime in your db of when your tweet was retweeted.

In this case, you would have a syntax like:

    result = my_schedule_module.retweet.apply_async( (tweet,) , link=my_schedule_module.callback_to_store_results_of_retweet.s())

Thank you for that, but I have been trying celery before. I have encountered problem with request context: RuntimeError: Working outside of request context. This typically means that you attempted to use functionality that needed an active HTTP request. — Lucas, Dec 05 '16 at 14:21
Further, what does it mean in practice? CELERYD_TASK_TIME_LIMIT = 600, CELERYD_TASK_SOFT_TIME_LIMIT = 600, CELERYD_MAX_TASKS_PER_CHILD = 1000, CELERYD_OPTS="--time-limit=600 --concurrency=4", - that retweet process can last max 600 seconds and then terminates? — Lucas, Dec 05 '16 at 14:26
Regarding my first comment on request context - after all tweets to be scheduled or words/hastags to be retweeted comes from user inputing them at the html form. I would use celery, but I do not know how to apply celery with html form context. — Lucas, Dec 05 '16 at 14:39
Celery is a proper approach for background jobs. http://stackoverflow.com/a/32890603/305883 for running application outside of context - check how you import modules: I put the celery_app in a separate module, and import that in Flask_app. You may design like user inputs in the html form, store data to mysql, and asyncrhonously you processed them. Use a second db to handle the queue (e.g. redis). Here you find docs for configuration: http://docs.celeryproject.org/en/3.1/configuration.html. Concurrency is the number of max workers to set up (as example, if you want to parallelise jobs). — user305883, Dec 05 '16 at 16:09

Twitter, multiple processes and database

1 Answers1