-1

I have a little pet project with a friend where we're looking to create a program for real-time visualisations, mainly utilising audio/video material, controlled by MIDI. Now, the MIDI part is not a problem, you seem to find decent solutions for almost any language, but I find myself terribly unsure of which way I should look regarding handling the video in a smart way. I'm looking for both fast seeking and additional visualisations (multiple superimposed pictures for example).

I have already experimented with a couple of options that I found were extremely easy to play with and seemed to offer at least something for the task, but with each of them I felt I might run into dead ends or low performance later on when looking to add features etc. So far, I tried Pure Data, Max and Processing.

What I'm mostly asking advice for is to direct me onto an optimal or at least a decent path regarding dealing with the videos. The biggest problem is I find myself using all my time only trying to find out what programming language or library I should use. If I only got that much guidance I could finally start really working on it and advance.

I suppose I'm most comfortable with python but any suggestions are welcome. I have read a little about gstreamer and I'm thinking there might be something there, but now we're talking about a relatively low level library that would take at least some time to produce any results with, as opposed to Processing or Pure Data/Max, for instance.

In addition to the language/library I'm curious about the importance of the video format. It goes a little beyond me when we start talking about codecs, I-, P-, B-frames and whatnot. Who knows, there could even be a solution where we'd use an optimal video format, cram that baby into RAMdisk or something and get satisfactory seek speed with only that.

umläute
  • 28,885
  • 9
  • 68
  • 122
vekkuli
  • 9
  • 2
  • 1
    Welcome to SO. This kind of question format is not really what the site is geared towards. There are many options to approaching this kind of project - including in Python, Java/Processing etc - and you're probably best places to decide which is most in line with your skills. In my experience Processing/Java is most powerful for complex media rendering, but others would no doubt disagree. – geotheory Aug 26 '13 at 01:09
  • Huh, point taken. Well thank you very much for answering in any case. – vekkuli Aug 26 '13 at 05:32

3 Answers3

1

I recommend having a look at creative toolkits and for practical reasons choose what language your prefer:

If language isn't an issue, maybe go ahead with MaxMSP and Jitter. Max has objects for handling midi input and parsing it. Jitter deals with fast matrix operations(images/video/gl/etc.) I think the equivalent for PureData is Gem

From experience, Java's garbage collector sometimes has the nasty habit of getting in the way when dealing with realtime audio, but that's mostly true for synthesis. You should be fine with MIDI. Depending on the resolution/amount of content the video part might get a bit slow.

Which is why I recommend either going straight for a c++ creative toolkit or Max. If you're not sure about video content, need many tweaks/iterations and fast changes, maybe you should go straight for Max which is easy to prototype in/pretty fast and you can pretty much 'live code' adapt you patcher while the code is executing and update/adapt/improvize based on your needs.

George Profenza
  • 50,687
  • 19
  • 144
  • 218
  • Good then, I suppose the way to go for now is to keep studying Processing, pd and MaxMSP first and see how those will turn out. The VVVV and TouchDesigner sure look worth giving a quick try at least. The C++ OpenFrameworks looks viable as well, if I encounter any major limitations with the "easy" ones I shall look into it. – vekkuli Aug 28 '13 at 07:59
  • I assume that depends totally on your setup - what is your output ?(e.g. will it be a projector ? if so, what resolutions does it support ? etc.) My guess is it will come down to a balance between speed/resolution/compression: if you're working with an extremely massive resolution it might interfere with the real-time side of things, if it's too small it won't look good. – George Profenza Aug 28 '13 at 10:27
  • We might use projectors but I think we're still talking about relatively low resolutions, say below 720p at least. I would even go and say the image quality comes last, response time and audio/video sync first. – vekkuli Aug 28 '13 at 10:58
1

my guess is that the problem is not MIDI input. most languages do support midi devices even home made ones. i guess its better to check out more specific software that deal with real time video/manipulation/projection

one which is kind of easy is VVVV http://vvvv.org/

another one is TouchDesigner used in Amon Tobin's ISAM shows (check them out) http://createdigitalmotion.com/2011/08/amon-tobins-isam-behind-the-scenes-ask-your-questions/

nowhere
  • 11
  • 3
0

personally i would suggest using a cross-platform (at least w32, linux, OSX) FLOSS solution. if you ever want to change your OS, you can pretty much rule out VVVV and max/jitter.

i would suggest to either use:

  • Pure Data / Gem for a graphical patching system

  • Processing if you prefer a traditional, text-based approach to programming (Processing is really just a front-end to java)

note, that personally i am highly biased towards Pd.

umläute
  • 28,885
  • 9
  • 68
  • 122