5

let's consider a variation of the "WAV to MIDI" conversion problem. I'm aware of the complexity of such a problem and I know that a vast literature about the more general Music Information Retrieval (MIR) subject exists.

But let's suppose here that we already have both the WAV and the MIDI representation of a music piece, so we actually don't have to discover pitches inside the WAV signal from scatch... we "just" have to match the pitches detected (using a suitable algorithm) with the NoteOn events contained in the MIDI representation. I definitely suppose we should use the information contained in the MIDI file to give some hints to the pitch detection algorithm.

Such a matching tool could be very useful, for example for MIDI "humanization": we could make the MIDI representation more expressive using the information retrieved from the WAV signal to "fine tune" note onsets, durations, dynamics, etc...

Does anybody know if such a problem has already been addressed in literature? Any form of contribution or assistance will be greatly appreciated.

Thanks in advance.

skaffman
  • 398,947
  • 96
  • 818
  • 769
C. Fischetti
  • 95
  • 3

2 Answers2

1

At the 2010 Music Hackday in London some people used the MATCH Vamp plugin to align score to Youtube videos. It was very impressive! Maybe their source code could be of use. I don't know how well MATCH works on audio generated from MIDI files, but that could be worth a try. Here's a link: http://wiki.musichackday.org/index.php?title=Auto_Score_Tubing

This guy appears to have done something similar: http://www.musanim.com/wavalign/ His results are definitely interesting.

Andreas Jansson
  • 3,137
  • 2
  • 30
  • 40
0

This seems like an interesting idea. What are you trying to do, is it just match the notes pitch? Or do you have something else in mind?

One possible thing that you could look into is if you know the note (as an integer value I think its been a while) that will be used to pass into the noteOn method, you may be able to do something with that to map it with a wav signal. IT depends on what you are trying to do.

Also, there are some things that you could also play around with in (I think it is called) the midi controller. Such as: modulation, pitch, volume, pan, or play a couple of notes simultaneously. What you could do with this though, is have a background thread that can change some of those effects as the note is being played. For example, you could have a note get quieter the longer it is played, or have a note that with pan between the left and right speakers, etc

I havnt really played with this code in a long time, but there are some examples of using a midi controller.

John Kane
  • 4,383
  • 1
  • 24
  • 42
  • I'll try to be more specific. First of all, I'm not looking for a realtime solution: the matcher would run in batch. In addition, let's concentrate on a single instrument (say Piano)... but the algorithm should be able to match polyphonic music. The idea is to export a "mechanical" MIDI execution from a score notation software (say MuseScore) and then "humanize" it by comparison with a human WAV recording. The point is that MIDI is much more handy than WAV, especially from an educational perspective, but usually much less expressive: I'm trying to improve its expressivity. – C. Fischetti Jun 10 '11 at 22:58
  • what type of data would you have from a MIDI execution? is it just the recorded sounds? OR would you have the music notation? There are other ways to play sound other than MIDI. What language would you be doing this in. I know in java there is a sampled sounds api as well as midi. – John Kane Jun 13 '11 at 12:20