Transharmonics

I got to thinking: what would a glissando sound like if the individual partial tones did not move in parallel? I should unpack that question a bit. A glissando is a smooth, continuous change in musical pitch, like a trombonist moving the slide while playing, or a violinist sliding a finger along the fingerboard. Most keyed or valved instruments are capable only of approximating glissandi through rapid chromatic scales, but for the purposes of this discussion I’m taking about a truly smooth, continuous change in pitch. The other part of my question that deserves explanation is the idea of partial tones. There’s a basic theorem in signal processing that tells us that any oscillatory signal (like the oscillations on your eardrum that your brain interprets as sound), can be understood as a combination of many very simple oscillations. Mathematically, these simple oscillations are called sinusoids, like the sine and cosine functions that you once learned about in a trig class. The individual sinusoids are sometimes called the partial tones, that is, the simple oscillations (each characterized only by a single frequency and amplitude) that combine together to form the complex oscillatory signal that rings our eardrum. Much of the timbre of a sound, that is, much of how our brains can tell the difference between a violin playing A440 and a trumpet playing A440, is encoded in the structure of the partial tones. Signal processing people, and some musicians, refer to this partial tone structure as the spectrum of a tone, analogous to how white light is a combination of many different kinds of colored light, each with different frequencies. Each of those colors is a “partial tone” in the spectrum of the white light. In fact, if you combine all audible frequencies in a single sound (incoherently), you end up with a sound called white noise, like some people use to help them get to sleep.

The idea of a glissando, and of partial tones, can be visualized pretty well with the help of a movie:

This video shows a kind of graph that we call a spectrogram, which describes how the partial tone information varies over time. The horizontal axis traces out time, the vertical axis represents frequencies, and the bright orange curves are the partial tones. The brightness (in this case, “orangeness”) of any pixel of the image tells us how strong the sinusoidal oscillations are at the given frequency (vertical axis), at the given time (horizontal axis).

Side point, slightly technical but super-cool: there’s a bit of mathematical subtlety in this description, because the partial tones themselves are oscillations that play themselves out over time. So strictly speaking, you can’t really say how strong the, say, 895-Hz oscillations are at precisely the 15 second mark, because you’d actually have to allow that oscillation play itself out over at least one 895th of a second before you really know what kind of oscillation it was. This subtlety means that you can’t construct a perfectly “fine-grained” spectrogram, there’s always some blockiness in both the time and frequency dimensions, pixelation of the image. You can see some of this pixelation quite clearly in the spectrogram above. The reason I went off on this tangent is because it actually transforms itself into something really profound in physics: the Heisenberg Uncertainy Principle. The uncertainty principle is often described intuitively as “you can’t measure a system without disturbing it.” That adage is true, kinda, though a little tricky to define rigorously. It isn’t exactly the uncertainty principle as physicists really know it. A better intuitive description is: “the better you know the location of a particle, the less well you know its momentum, and vice versa.” This relates back to signal processing because mathematically we can think of the spatial location of a particle as analogous to the time (horizontal axis) in the above spectrogram. If we do so, then the standard mathematical formalism of quantum mechanics tells us to think of the particle as a kind of wave, that oscillates over space, and to describe the particle’s momentum as being analogous to the frequency (vertical axis) in the spectrogram (technically, physicists call the ‘frequency’ of oscillations over space the ‘wavenumber’, instead of the frequency, but that’s way off topic). The necessary blockiness of the spectrogram is, literally, the uncertainty principle! The thinner the pixels are, the taller they’re forced to be (the better you know the position, the worse you know the momentum); the shorter the pixels are, the wider they’re forced to be (the better you know the momentum, the worse you know the position). Anyway, back to sound.

In the above spectrogram, you’ll see sixteen continuous curves. These are the sixteen individual partial tones that I combined together to synthesize the kinda casio-keyboard-oboe tone that I ended up with. Note that the curves are equally spaced. At the left end they’re each stacked exactly 165 Hz above the one below. Turns out this even spacing is necessary for our brains to assign a definite pitch to the combined oscillation. That pitch is, as you’d probably guess, the the pitch that we’d associate with a 165-Hz sine wave, namely E3 (the third E up from the bottom of the piano keyboard). At the right end of the image the curves are evenly spaced again, now each spaced 110-Hz above the one below, meaning we hear it as the pitch A2. In between, the partial tones all transform themselves in such a way that they continue to be evenly spaced, but the spacing is slowly decreasing, meaning the pitch we hear goes down. In the movie, I’ve drawn a thin vertical bar to represent “current time.” It probably isn’t all that necessary for following the simple glissando above, but things are about to get more complex.

So, as for my question: what if the partial tones don’t move in parallel? That is, what if we start with the same 165-Hz tone, progress to the same 110-Hz tone, but in between we allow the partial tones to mix themselves up in a more complicated way? The quick answer is that our brain relies on the even spacing of partial tones to infer the pitch of a sound. So whenever the partial tones are not evenly spaced, our brains will struggle to assign a musical pitch to whatever’s going on. The brain might break up each partial into individual oscillations, each heard as “separate” sounds. Sometimes it might find enough approximate structure to assign a combination of two or three individual pitches (the brain does this, to some extent, when hearing some kinds of bells or drums, which don’t produce evenly-spaced partials), or the brain might just not be able to settle at all on what’s going on, and the result might simply be classified as “noise.”

But what does it sound like? That’s what I was wondering.

Well, here’s an example. I may or may not be the first person to come up with this kind of signal, but being a bit optimistic I’ve given it a name: a transharmonic glissando. The even spacing of partial tones, which our brain needs to have to assign a pitch, is called a harmonic sequence. The transharmonic glissando transitions from one harmonic sequence (the one associated with 165 Hz) to another (110 Hz), but during the transition, harmonicity is lost, leading to much more complex auditory structure. Specifically, I refer to this as a “shuffle glissando”, because the individual partials are shuffled during the transition. I’ve also significantly lengthened the transition to allow time to hear all the cool stuff that goes on:

A few things you might notice: at least to me, the audio sounds louder during the transition. It isn’t (at least, the amplitudes have not been artificially strengthened). I’ve found this effect to be stronger with rapid transhamonic glissandi, but I’m pretty sure I’m hearing it here. The added loudness is entirely in your mind, because before and after the transition your brain hears only one pitch, and during the transition it’s forced to contend with many. There’s some really fascinating psychoacoustics that goes into precisely what we mean by “loudness”, which I shouldn’t go into here, but this might at least be a demonstration that the mind is an easy beast to confuse.

Also, anyone who’s ever put much effort into tuning an instrument will notice the “beats” and “pitch fusion” that occur whenever the individual curves cross each other, especially the lower ones for which beating tends to be more obvious.

Another thing that I’m noticing, and I have no idea if this is a known (or even real) effect. My brain holds on to the idea of E3 (the first pitch) pretty well into the transharmonic period. The pitch stops being E3, strictly speaking, at the 5-second mark, but my brain hears it as at least “funny-sounding E3” until nearly the 10-second mark. The return to harmonicity at the end (the A2), is much more sudden, at least to me. It doesn’t particularly surprise me that the brain will “hold on to” whatever symmetry it can find for quite a while even after the symmetry is broken, while recognizing the development of new symmetries is a much more sudden process.

So that’s the basic idea of a transharmonic glissando. First there’s harmonicity (pitch), then anharmonicity (“noise”), and eventually the harmonicity returns. The main moral of the story: even though each element of the tone is transformed in a continuous way, there is nothing continuous about the pitch. The whole idea of pitch is completely lost when the partials are no longer evenly spaced.

This got me to thinking: is there a more gradual way to do all this? Is there a glissando that transitions from one pitch to another, again in a gradual and transharmonic way, that preserves some semblance of pitch? The best way that I’ve come up with so far is something I call the cascade glissando, wherein each partial transitions as in an ordinary glissando, but they do so at separate times; specifically, in succession. Here’s a realization of an “upward cascading glissando”, again from E3 to A2 (so a musician might be stuck calling it a “descending upward cascade glissando”):

Again, the transition here is anharmonic, but it’s carried out in a systematic-enough way that the mind is able to follow along and hear the E3 “slowly disappearing” and the A2 “slowly taking over.” Note also that in this case, the partials don’t happen to cross each other, so there is no significant beating or pitch fusion going on.

What if we do the same thing, but start with the “top” partials and work our way down? Well that’s a good question! Thanks for asking! The result is what I’m calling a “descending downward cascade glissando”:

Notice that just due to the basic geometry of the process, the partials are forced to cross each other. In fact they do so quite often. This leads to a much harsher sound, especially near the end, when beating becomes very audible.

Finally, a middle ground between these two. What if we again let these partials transition one-by-one, but we choose the ordering randomly instead of ascending or descending? The result is what I call a “descending random cascade glissando.”

In exploring these questions I built up quite a substantial computer code for synthesizing these effects. At this point, I actually think of this code as an instrument in itself, the transharmonic organ. Clearly the project has gotten my creative juices flowing and if all goes well it will get me composing music again, which I haven’t really seriously done in about fourteen years. Back at that time, when I was pulled away from music by the demands of grad school, I was fascinated with the rhythmic structures that could arise “accidentally” from broken harmonic structures. I would take short melodic phrases, of differing lengths, repeat them, and combine them in canon. The differing lengths of the phrases caused them to shift in relation to one another, building up a process that takes many repetitions to work itself out. Along the way, the standard rules of counterpoint were at times blatantly violated, leading to jarring dissonances that themselves provided a new layer of rhythmic structure. (Yes, I was a big Steve Reich fan. Still am.) In exploring these anharmonic transitions, I’ve found a “microscopic” version of this rhythmic dissonance, which could be very fruitful musically (artsy-fartsy music, granted, but music nonetheless). I understand that spectral structure has been a big fad in the compositional world for the last few decades, so maybe this idea is nothing new, but I’m really interested in exploring it.

In particular, the tones that I’ve been playing with so far are monophonic, as in, before and after the transharmonicity, there are only single pitches playing. This condition is not at all necessary, and in musical applications I could in principle work out polyphonic music in which individual partials are exchanged between the separate voices. The structure of the music would be very different from conventional polyphony, because I’d want to take each transition extremely slowly to allow the listener to comprehend the transharmonic effects. I’m reminded of a very old Steve Reich piece, called “Four Organs”, which I believe he once described as “a fifteen-minute-long 5-1 cadence.”