Can this software create 3D meshes from a MIDI file?

[sebarious70]

I'm trying to create a 3D scene from a midi file. The scene would need to have each note as a cube rectangle, and the translation from the music world into the 3D scene would be basic: 1 bar = 1 meter. So if you have a MIDI note that is 1 bar long, the long side of the rectangular cube would be 1 meter long. The other two dimensions would be 25 cm each. So, something like this:

I'm on the verge when it comes to the other two dimensions for shorter notes, because if I keep 25 cm for all of them, when I do notes that are 1/2, 1/4 and 1/8, it would look like this:

But if I also shorten X and Z more or less proportionally, I think perhaps it would look better:

Regardless, my goal is to get something like this:

into something like this:

Obviously without the flat notes, those are just to show the difference between the flat ones and how the 3D ones should be)

My secretary (that would be the free version of Chat GPT) pointed me to this excellent software when I asked it if there was any program that can generate 3D meshes from MIDI notes. So I started reading the documentation and I found some hope, but first I want to ask here if what I'm trying to achieve is possible with this add-on. I don't even need animation, just static notes, all laid out in a 3D scene.

Now, there's a section of the documentation that I spent a lot of time trying to build exactly as it shows, but I didn't get anywhere near the animation that shows there, or saw any meshes being built: . The only thing I achieved was to move the first cube I created in -Y.

Now, I copied the node tree under "Raw Note Information" as best as I could, but there is one node, the group input all the way to the left that has two inputs, one is a MIDI track, and the other is Width. Well, "Width" is not a name that shows up when I click on the + and type width. In fact, it doesn't appear in anything. Either the documentation is outdated or Blender 4.1.0 has something so it doesn't show? I couldn't say. But other than that, I think all the rest are just like the example in the link above:

So that's all. Basically I want to know if this add-on can be setup in a way that will ingest a MIDI file (and it won't be a super complicated one, just one track), and turn that into a 3D scene, and if so, if this is more or less the node setup I need, or if I need to go another route.

[OmarEmaraDev]

The Width input is just a float input whose name is Width. You can definitely create what you want using this software, I can provide an example file if you couldn't get it to work eventually.

[sebarious70]

Good, at least now I know it's possible. Now, what's the learning curve? Because what I need to do can be done manually by setting the viewport to top view and putting a long wide screenshot of the section of the song that I want to do this for as a background, then grab the 1 bar note, duplicate several times, then the same thing with the other lengths, and finally placing them to match the notes. Time consuming, but not terribly so.

On the other hand, if the automation nodes way means spending days learning the stuff, that's time that I don't have. Would love to, because it looks like a great thing to learn, but I don't have the luxury of time.

Basically the thing I study music, and I have some songs that I would like to upload to YouTube, but with visuals, and I have a waveform that I did in Fusion with a 3rd party plugin, but then I thought about adding a way to visualize parts of the song with a MIDI track in 3D. But this is something I want to upload as soon as possible, so I can't spend a week or two learning this.

How difficult is it to get it to create the meshes from a simple MIDI track, only index 0? I've been trying some things this afternoon but I couldn't find the way to actually generate meshes from the MIDI track.

[OmarEmaraDev]

@sebarious70 I am not not sure about the learning curve, because it likely depends on how familiar are you with Blender, nodes, maths, and so on. But this particular thing you want to do is rather easy, so it is probably worth learning a bit, and we can help you along the process.

I made you a simple example to get you started:

Just change the path to your MIDI file and you should get MIDI notes as meshes. The Get List Element retrieves the first track from the file. Then we get the note info of that track, and compute the note length by subtracting the on time from the off time. The X position of the note is its On time, its Y position is its note number. The X scale of the note is the note length, while its Y and Z scales are fixes, which you may change based on some note info.

midiToMesh.zip

[OmarEmaraDev]

@sebarious70 It seems you don't have the Object Instancer node selected. Those seems like the options for another node.

[sebarious70]

@OmarEmaraDev thanks, now I see it. But I also discovered a better way, and I may be able to keep the node tree that separates each note as a different object, because Node Tree settings in the N panel have an Auto Execution check that obviously was punishing Blender because it was trying to build 1576 cubes constantly, while if that's disabled, then it's 1576 objects but they are static. This way it works out better because if I happen to change something, then I wouldn't want to build the whole node tree again. Then if I want to rebuild, I press the "Execute node tree" button.

Although I would have to see if once I add the other tracks that I plan on adding, and this amounts to over 10,000 cubes, if Blender is still going to be responsive having that many objects in the scene. So I guess I'll have to try that.

BTW, is there a way to save a node tree to use in another project? I spent a lot of time working on the other project that had all the notes as one mesh, so I have a lot of animation stuff (not for the animation nodes, but Blender native position and materials, etc) that it would take me too long to bring back into this project, so I would to recreate this node tree in the other project, but I've been wondering if you can just save the node tree to a file, or something like that.

[OmarEmaraDev]

@sebarious70 You can just append the node tree to another file. If you wrap your node tree into a subprogram, the subprogram will be accessible in all node trees. Otherwise, use the appended node tree directly.

[sebarious70]

@OmarEmaraDev I keep going down the rabbit hole in this project, and I didn't ask you this before because I thought it might be out of scope for AN, but today I was thinking that perhaps it's not. The next step for me would be for the note that is playing currently to be lit up. Let me show you with screenshots.

Right now I have a wireframe modifier to the Target object, and two materials assigned to it, the first slot is a plain black material, the second one is an emission material, so basically this:

I have the camera keyframed to travel from the beginning of the MIDI track to the end. What I want to accomplish is that the note that is playing currently changes from having emission only in the wireframe to have it in its whole surface. To give you an idea, and this is just a mockup:

I've been trying to figure that out for several days if not weeks, and no matter how much I search, or read the manual, I can't find the answer. It's easy to get them all lit up or not with geometry nodes:

But, having just one go into full mode sequentially, that's the challenge. So far I had been focusing on the nodes in the shader, then geometry, because I figured AN built the meshes but didn't have much to do with materials.

But then one thing occurred to me; what if AN can assign some type of ID to each cube it duplicates, and that's an ID that somehow can be picked up by geometry nodes so when I change the Set Material Index it only applies to that specific cube, not to all of them.

Does that sound like something that is possible?

[OmarEmaraDev]

What you are describing seems like what I demonstrated in #1962 (reply in thread) already?