0

I'm relatively new to Python and Flask. I've been trying to create a web application that reads data readings from a .txt file and plots them onto a matplotlib plot. In this web-app, I have a front page with 3 buttons. These buttons redirect to different routes with functions that read data and plot them onto a matplotlib plot. The web-app works perfectly only for the first time I go to either of these routes. After that nothing loads anymore. I guess I have an infinite loop of some sort but I can't figure it out. Also, after the website gets stuck the Python process starts consuming more resources.

The problem persists only when I open this route on the web-app:

@app.route("/temperature/")

This loads without problems on the web-page, but only for one time and the whole web-app gets stuck and I cannot access any of the other routes either.

Thanks in advance!

EDIT 1 - Whole code below

cloudapp.py (Python sourcecode that runs Flask and the functions)

from flask import Flask
from flask import render_template
from flask import request
import numpy as np
import matplotlib.pyplot as plt, mpld3
from datetime import datetime

app = Flask(__name__, template_folder='C:\Users\Valtteri\Desktop\cloudapp\Templates\HTML')
global all_lines

@app.route("/")
def frontpage():

    return render_template('frontpage.html')


@app.route("/temperature/")
def temperature():


    f = open('C:/Email/file.txt', 'r') 
    cpt = 0 # Line amount value
    all_lines = [] # List that has every Nth value 
    for line in f:
        cpt += 1 # Goes through every line and adds 1 to the counter
        # How often values are plotted (every Nth value)
        if cpt%100 == 0: 
            all_lines.append(line) # When 30th line is counted, add that line to all_lines[] list
        if cpt == 500: # How many values are plotted (counts from first)
            break

    dates = [str(line.split(';')[0]) for line in all_lines]
    date = [datetime.strptime(x,'%Y.%m.%d_%H:%M') for x in dates]
    y = [float(line.split(';')[1]) for line in all_lines]
    z = [float(line.split()[2]) for line in all_lines]

    fig, ax = plt.subplots()
    ax.plot_date(date, y, 'r-')

    f.close()
    return mpld3.fig_to_html(fig)
if __name__ == "__main__":
    app.run()

frontpage.html (HTML template in folder ..\templates\html)

<!DOCTYPE html>
<html>
  <head>
<meta charset="UTF-8">
    <title>Envic Oy Cloud</title>
  </head>
  <body>
  <div class="headers">
  <h1>Web-Application</h1>
  <h2> Version 0.0.1 </h2>
  </div>
  <div class="buttons">
  <h3 class="buttonheader">Logger 1</h3>

  <a class="templink" href="http://127.0.0.1:5000/temperature/" target="_blank"> Check temperature </a>
  </body>
</html>

I use Bash on windows to run the code with

export FLASK_APP=myCloud.py

flask run

EDIT 2

I tried to solve the issue for a long time, but couldn't find a solution. It has something to do with Flask/mpld3 compatibility for me. I made the very same web app, but this time using a simple pyramid WSGI. I can now refresh the plot as many times and redirect myself into any view without the server hanging. I'll still leave the post unsolved, because I would still like to use Flask. I will continue my research too. Here is the pyramid version that works for me:

from wsgiref.simple_server import make_server
from pyramid.config import Configurator
from pyramid.response import Response
import numpy as np
import matplotlib.pyplot as plt, mpld3
from datetime import datetime

def hello_world(request):
    return Response('<h1>Testing the Pyramid version!</h1><a href="http://localhost:8888/second_view">Check temperature</a>')

def second_view(request):
    with open('C:/Email/file.txt') as f:   

        cpt = 0 # Line amount value
        all_lines = [] # List that has every Nth value 
        for line in f:
            cpt += 1 # Goes through every line and adds 1 to the counter
            # How often values are plotted (every Nth value)
            if cpt%100 == 0: 
                all_lines.append(line) # When 30th line is counted, add that line to all_lines[] list
            if cpt == 500: # How many values are plotted (counts from first)
                break

        dates = [str(line.split(';')[0]) for line in all_lines]
        date = [datetime.strptime(x,'%Y.%m.%d_%H:%M') for x in dates]
        y = [float(line.split(';')[1]) for line in all_lines]
        z = [float(line.split()[2]) for line in all_lines]

        plt.figure(figsize=(10,5))
        plt.title('Humidity', fontsize=15)
        plt.ylabel('Humidity RH', fontsize=15)


        fig = plt.figure()
        plot = plt.plot_date(date, z, 'b-')


        myfig = mpld3.fig_to_html(fig, template_type='simple')
    return Response(myfig)

if __name__ == '__main__':
    config = Configurator()
    config.add_route('hello_world', '/hello_world')
    config.add_route('second_view', '/second_view')
    config.add_view(hello_world, route_name='hello_world')
    config.add_view(second_view, route_name='second_view')
    app = config.make_wsgi_app()
    server = make_server('', 8888, app)
    server.serve_forever()
  • Try unindenting your `return` statement. My hunch is that the file handler never closes the file so you probably have a file lock on the file in subsequent reads. – Scratch'N'Purr Jul 02 '18 at 06:59
  • Thanks for the response. Tried that, but unfortunately didn't solve the issue. I also tried to do f.close() before the return statement, but it still remains the same. I am taking the same guess as you, that it is the file handler that doesn't close and the code cannot process anything else. – DeersAreFriends Jul 02 '18 at 07:21
  • The issue seems to be with the plot itself (matplotlib or mpld3), see the edit in my original post. – DeersAreFriends Jul 02 '18 at 07:33
  • Hmmm... I can't seem to replicate your problem. However, while playing with your code, I noticed you have 2 `plt.figure`s. I believe you want to remove the 2nd one and set `fig` to the 1st statement. I also tried using `plt.clf()` and `plt.cla()` right before the `return` statement to see if pyplot can flush its cache. – Scratch'N'Purr Jul 02 '18 at 08:44
  • Interesting. I now have only one route with only a simple code that creates a plot with matplotlib and prints it on the web page with mpld3, and I still have the issue. It loads once, but after that it becomes unresponsive and the page can't be even refreshed. The loading icon appears on chrome but it just keeps spinning forever. – DeersAreFriends Jul 02 '18 at 09:23
  • I might have figured it out. MPLD3 starts its own web server at 127.0.0.1:8888 and my Flask application runs at 127.0.0.1:5000. Anyway to fix this? – DeersAreFriends Jul 02 '18 at 10:16
  • I'm not sure why you're running the MPLD3 server. The `fig_to_html` function should only return the HTML code which gets parsed in the browser to display the chart. There shouldn't be any other server running except for Flask. – Scratch'N'Purr Jul 02 '18 at 11:27
  • You are right, realized this 5 minutes after my comment. I still have the same issue. Main page has the link to open '/temperature' route, everything goes fine, that opens in a new tab like it should and it loads fine, I can interact with the chart. I can go back to the main page and refresh it, everything still fine. HOWEVER when I try to open/refresh the plot again, everything gets stuck. Or if I try to open another route with a plot. Seems really weird and can't wrap my head around this. I have tried a lot of things but nothing seems to have solved it yet... – DeersAreFriends Jul 02 '18 at 11:36
  • I'm not able to replicate your issue. I'm using the same code you have, except for the couple edits that I stated earlier. I can refresh the route without any blocks. Do you have a `plt.show()` somewhere in your code? – Scratch'N'Purr Jul 02 '18 at 11:41
  • No I don't. I'll edit my whole code to the post including the HTML template so you can try it out 100% and see if it works. Then it might be an issue with my browser/PC. Thanks for helping :) – DeersAreFriends Jul 02 '18 at 11:59
  • Updated the post, feel free to try the code out again. I would appreciate the help! – DeersAreFriends Jul 02 '18 at 12:14

1 Answers1

0

Here is what I attempted. I didn't get any issues with the server hanging after refreshing or opening a new tab of the link. However, after I shutdown the server (Ctrl + C), the console threw some exceptions that suggests that mpld3 or matplotlib had opened some new threads. Specifically, the exception is RuntimeError: main thread is not in main loop.

I did some googling and came across this link. The guy suggested using fig_to_dict with json. I tried his solution and I was still getting the exception.

Now, I'm going to write down both approaches and let you decide which to use. I don't know if either of them will work for you. For my setup, the app runs fine despite the exceptions after closing the server.

I'm also going to use your example that didn't read the txt file. I have debug set to True so I can make sure the GET requests are being processed when I refresh the chart or open a new instance of the link.

Approach 1 (fig_to_html)

app.py

from flask import Flask
from flask import render_template
import matplotlib.pyplot as plt
import mpld3


app = Flask(__name__, template_folder='/path/to/templates')


@app.route("/")
@app.route("/index")
def index():
    return render_template('frontpage.html')


@app.route('/temperature')
def temperature():
    date = ([1, 2, 3, 4])
    y = ([1, 2, 3, 4])

    fig = plt.figure(figsize=(10, 5))
    plt.title('Temperature', fontsize=15)
    plt.ylabel('Temperature' + u'\u2103', fontsize=15)

    plt.plot(date, y, 'b-')
    plt.ylim([0, 40])

    myfig = mpld3.fig_to_html(fig, template_type='simple')

    plt.clf()  # clear figure
    plt.cla()  # clear axes
    plt.close('all')  # close all figures

    # Print as HTML
    return myfig


if __name__ == "__main__":
    app.run(debug=True)  # run on debug mode

Approach 2 (fig_to_dict)

app.py

from flask import Flask
from flask import render_template
import matplotlib.pyplot as plt
import mpld3
import json


app = Flask(__name__, template_folder='/path/to/templates')


@app.route("/")
@app.route("/index")
def index():
    return render_template('frontpage.html')


@app.route('/temperature')
def temperature():
    date = ([1, 2, 3, 4])
    y = ([1, 2, 3, 4])

    fig = plt.figure(figsize=(10, 5))
    plt.title('Temperature', fontsize=15)
    plt.ylabel('Temperature' + u'\u2103', fontsize=15)

    plt.plot(date, y, 'b-')
    plt.ylim([0, 40])

    single_chart = dict()
    single_chart['id'] = "temp_figure"
    single_chart['json'] = json.dumps(mpld3.fig_to_dict(fig))

    plt.clf()  # clear figure
    plt.cla()  # clear axes
    plt.close('all')  # close figure

    # Print as HTML
    return render_template('temperature.html', single_chart=single_chart)


if __name__ == "__main__":
    app.run(debug=True)  # run on debug mode

And here are the template files. I noticed you used a static link in your frontpage.html so I replaced that with a placeholder that allows Flask to auto-populate the URL. You also had a div tag that you didn't close.

frontpage.html

<!DOCTYPE html>
<html>
  <head>
    <meta charset="UTF-8">
    <title>Envic Oy Cloud</title>
  </head>
  <body>
    <div class="headers">
      <h1>Web-Application</h1>
      <h2> Version 0.0.1 </h2>
    </div>
    <div class="buttons"></div>
    <h3 class="buttonheader">Logger 1</h3>
    <a class="templink" href="{{ url_for('temperature') }}" target="_blank"> Check temperature </a>
  </body>
</html>

temperature.html (for 2nd approach only)

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <title>Sample Page</title>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/require.js/2.1.10/require.min.js"></script>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/2.0.3/jquery.min.js"></script>
    <script type="text/javascript" src="http://d3js.org/d3.v3.min.js"></script>
    <script type="text/javascript" src="http://mpld3.github.io/js/mpld3.v0.2.js"></script>
  </head>
  <body>
    <div id="{{single_chart.id}}">
  </div>
  <script type="text/javascript">
    var figureId = "{{single_chart.id}}";
    var json01 = {{single_chart.json|safe}};
    mpld3.draw_figure(figureId, json01);
  </script>
  </body>
</html>

BTW, I'm using Python 3.5.4, Flask==0.12.2, matplotlib==2.1.2, and mpld3==0.3. I tested using Chrome Version 67.0.3396.87.

Scratch'N'Purr
  • 9,959
  • 2
  • 35
  • 51
  • Thanks for the help. I tried out your version, but unfortunately I still have the issue. I click the link, it opens and loads the plot. After I close the plot and try to open it again, it does not process 'GET' requests anymore. I am using python 2.7, could it be that? – DeersAreFriends Jul 03 '18 at 05:55
  • The fig to dict version did not work at all. It loaded up a blank page. After that the GET requests didn't work anymore again. This issue is so frustrating. I might try the code on another PC with python 3.5. Oh BTW, my program couldn't navigate to URLs with the route address @app.route('/temperature'), for me I need to put another forward slash on it to work ('/temperature/'). Otherwise it gives me an error that URL not found. – DeersAreFriends Jul 03 '18 at 07:09
  • I tried with 2.7 and didn't have issues either. I did some more googling and found this [repo](https://github.com/nipunbatra/mpld3-flask/blob/master/routes.py) that might be worth looking at. The guy creates a lock before creating his plot. If this doesn't work, then I can't really help you any further since I'm out of ideas. – Scratch'N'Purr Jul 03 '18 at 07:11
  • Okay. Seems like a really weird issue.. I can't thank you enough for your help, was really kind of you. Would have taken a lot of my time troubleshooting this and thinking that there is something wrong with my code. I'll try that one out, and if it does't work, i'll switch my PC and go with a fresh start. Hopefully I get it working :) – DeersAreFriends Jul 03 '18 at 07:37
  • Not a problem! Hope you get it working! I'd be interested to see the solution if you figure it out. :) – Scratch'N'Purr Jul 03 '18 at 08:16
  • I'll update the post once I get it working and notify you :) – DeersAreFriends Jul 03 '18 at 08:24
  • Found a solution, at least temporary for now! Updating it on the main post – DeersAreFriends Jul 03 '18 at 12:17
  • That interesting. So switching to Pyramid got it working? – Scratch'N'Purr Jul 03 '18 at 13:53
  • Yes, indeed. I still have to try the Flask program on my other PC to see if it has something to do with my computer. – DeersAreFriends Jul 04 '18 at 06:07