0

My question is about web application architecture.

I have a website, my users can upload files and from this files I need to create some kind of reports for users. When user upload file it stored on my server where website hosted. File path stored in Django model field. Worker is on another server and i need to get access to my database and procces that file. I know how to use django ORM itself without URLs and other parts of django.

My question: If I need to create workers on another server, which use different django models from my website i need to copy all of the models into every worker?

For example, one worker proccess file and it need models "Report" and "User". Other worker do other actions and need "User" and "Link" models. Everytime i change model in my main website i need to change same models in my workers, also different workers can have same duplicate models. I think it's not good from architecture point.

Any suggestions on how to organize my website and workers?

xelblch
  • 699
  • 3
  • 11
  • Without more background information why you have two servers in the first place this question is rather narrow. Any reason why you can't use celery? – Hedde van der Heide Apr 17 '13 at 08:57
  • My django-based website hosted on heroku, not on my personal VDS where i can setup django-celery and run workers with supervisor to keep them alive. Thats why i only upload file, store path information in my database and queue remote worker from IronWorker. My worker is a script which download that file, procces it and create some objects in database. And every worker do his own task and need different models, but sometimes same. – xelblch Apr 17 '13 at 09:04

3 Answers3

1

You say you're using IronWorker so just include the models you need in each worker in your .worker file. For example, let's say you have a worker called report_worker.py, the one that needs "Report" and "User", point to each of those in the report_worker.worker file:

file '../models/user.py'
file '../models/report.py'

Or since you're using Django, you might have all your models in models.py, so:

file 'models.py'

Then when either of those changes, just reupload the worker with the cli:

iron_worker upload report_worker

Then you can use the same models as what your app uses. Hope that helps!

More info on .worker files here: http://dev.iron.io/worker/reference/dotworker/

Travis Reeder
  • 38,611
  • 12
  • 87
  • 87
0

Why do you really need the exact same models in your workers? You can design the worker to have a different model to perform it's own actions on your data. Just design API's for your data and access it separately from your main site.

If it really necessary, Django app can be shared across multiple projects. So you can just put some generic code in a separate app (like your shared models) and put them in sourcecontrol. After an update in your main website, you can easily update the workers also.

tuvokki
  • 720
  • 1
  • 10
  • 18
  • If I understand you correct the solution is something like this? http://pastebin.com/ZPfamFG0 – xelblch Apr 17 '13 at 09:13
  • You should investigate [django-piston](https://bitbucket.org/jespern/django-piston/wiki/Home) for a means to buils (REST)API's. You can create an API to your website's Report Model that just returns the fields you need in your worker. – tuvokki Apr 17 '13 at 09:28
  • Thank you, going take a look on it. – xelblch Apr 17 '13 at 09:31
0

There are few interesting options.

As example, you can add additional reupload workers step for deploy process. It'll guarantee consistence between deployed application and workers.

Using own (rest) api is great idea, i like it even more than sharing models between different beings

thousandsofthem
  • 1,355
  • 10
  • 9