0

I'm currently building a selfhosted Vuejs webapp (with account logins). The webapp needs to be a Python webscraper GUI where my user has control over the Python scraper.

So for example, the user fills in an endpoint, starts scraper, view results, trigger a new more depth scraper, etc.

I have the Python scripts for scraping. And I have decided to go with VueJS + AWS cognito + Hasura for the frontend

I have difficulties understanding how to trigger the python scripts and dump the results in the database and show them to the frontend.

I do like the 3 factor approach:

enter image description here

The data from my scrapers can be many db entries, so I don't like to enter them in the database via mutations.

Do I have to make Flask endpoints to let Hasura trigger these webhooks? I'm not familiar with serverless. How do I make my Python scraper scripts serverless? Or can I just use SQLalchemy to dump the scraper results into the database? But how do I notify my frontend user that the data is ready?

user3411864
  • 624
  • 2
  • 12
  • 27
  • In addittion to Jesse Carter answer: "How do I make my Python scraper scripts serverless?" deserves a separate question. What is your script, what you've tried to make it serverless, the problems you encountered. – Alex Yu Sep 04 '21 at 07:29

1 Answers1

2

There are a lot of questions in this one and the answer(s) will be somewhat opinionated so it might not be the greatest fit for StackOverflow.

That being said, after reading through your post I'd recommend that for your first attempt at this you use SQLalchemy to store the results of your scraper jobs directly into the Database. It sounds like you have the most familiarity with this approach.

With Hasura, you can simply have a subscription to the results of the job that you query in your front end so the UI will automatically update on the Vue side as soon as the results become available.

You'll have to decide how you want to kick off the scraping jobs, you have a few different options:

  • Expose an API endpoint from your Python app and let the UI trigger it
  • Use Hasura Actions
  • Build a GQL server in Python and attach it to Hasura using Remote Schemas
  • Allow your app put a record into the Database using a graphql mutation that includes information about the scrape job and then allow Hasura to trigger a webhook endpoint in your Python app using Hasura Event Triggers

Hasura doesn't care how the data gets into the database it provides a ton of functionality and value even if you're using a different Database access layer in another part of your stack.

Jesse Carter
  • 20,062
  • 7
  • 64
  • 101
  • 1
    You are right, a bit too many questions for stackoverflow. Haven't found a good dev forum to discuss these kind of "stuck" moments. But your advice got me a lot further. I added an entire Flask api / gunicorn to my existing docker-compose. I can trigger the scrapers and let vue subscribe to the results – user3411864 Sep 04 '21 at 09:00