Guitar tuner that uses phone accelerometer
tautme.github.io171 points by adm4 9 days ago
171 points by adm4 9 days ago
My accelerometer apparently reads at 200Hz, but due to a lack of instrument at the ready I had to "pluck" the handle of the office fußball table.
When the right defender is near the center I'm reading ~24.74Hz, so slightly above G.
This was the direct inspiration for Total Football: to find harmony with the pitch.
heh, a subacoustic G#?
I think the tuner is off by an octave - it has a lot of harmonics and sounded like a G on a bass guitar.
It seems odd that it would be an octave too low. If it detected any harmonics it would be too high.
Unless it didn't function like a string at all, then the harmonics would be all wrong.
> If it detected any harmonics it would be too high.
I think it's not that simple. A tuner is "hearing" the fundamental and all of the harmonic overtones combined. It has to guess at which frequency is the fundamental, even if the overtones are actually stronger than it amplitude-wise, and it does that by looking at the nature of the repeating overtone pattern and extrapolating back to the fundamental.
I think you can end up an octave too low (half the actual frequency) if the waveform repeats in a way that implies a different overtone repetition pattern, for example if there's an every-other-cycle artifact to the waveform.
This has some very interesting privacy and security risks. If the tech can do more complex frequency analysis, then couldn't it essentially be used as a microphone for a device that doesn't need permission.
I thought this has been done to capture keystrokes of a keyboard next to the phone already
2011 https://www.researchgate.net/publication/221609349_spiPhone_...
It's a pretty well-known exploit that the CIA is capable of turning a lot of electronics with speakers into microphones. I imagine there is an entire classified backlog of things they can turn into microphones without the target's knowledge.
Tangent but the hidden no-electronics bug "The Thing / Great Seal Bug" really crazy
It had no internal power supply, it worked like an RFID tag, but it was electronic.
Tangentially, I didn't know this (from Wikipedia):
> The Thing was designed by Soviet Russian inventor Leon Theremin, best known for his invention of the theremin, an electronic musical instrument.
Yeah I was going for no battery
Idk though when I read it, it seems like it's literally an antenna attached to a can "resonator" is that electronics? It is I guess since it can carry an RF wave? Electronics I think of a chip or circuitry. I get it has to be some form of a circuit to work even as a monopole.
The article says it though: "...hung in his office behind his desk, and which contained an electronic device"
I think if you saw it on a bench hooked up to wires, it would be intuitive that it was a circuit. It's equivalent, but instead of being coupled via wires it's coupled via RF. I think it feels like there's no return path and that the circuit is open, but it's a real circuit with complicated/uncommon coupling to the power source.
A resonator is both a component in the circuit (the case is a cavity resonator) and the type of circuit this is. When illuminated (or hooked up to a power supply on the bench), it produces a sine wave, and holding all else equal the frequency is a function of the capacitance of our membrane capacitor. That membrane is flapping about due to sound, changing the distance between the plates of our capacitor and thus it's capacitance. So this shifts the frequency we're resonating at and encodes the audio into our output signal (frequency modulation).
So it's very similar to a standard LC resonator circuit you might make on a breadboard.
I'll leave you with another story of clever KGB sabotage. The KGB controlled facilities used to construct the US embassy in Moscow in 1979. They were able to extensively bug the building. They were also able to mix thousands of diodes into the concrete. This defeated NLJD (https://en.wikipedia.org/wiki/Nonlinear_junction_detector) based bug detection because they detected the diodes in every direction.
Thanks for the explanation and will look into NLJD, really need pictures, that bee is cool
The CIA…plug a set of regular headphones into a microphone jack, open a recording application and speak into the headphone speaker, you don’t need a 3 letter agency for that physics open secret.
I got this clone Apple lightning to 3.5mm headphone jack adaptor for iPhone, my mind was kinda blown when I found out it just uses the lightning for power and inside the plug is a tiny bluetooth device that stream music over the 3.5mm jack. The original adaptor doesnt work like this as far as i have considered.
Wouldn't you need to rewire the headphones? Headphones use a 3-pin TRS whereas a 4-pin TRRS plug is used when you add a microphone. Regardless if the 4-pin is CTIA or OMTP, it's generally only going to get shorted to ground if a 3-pin TRS plug is plugged into a 4-pin TRRS socket, or if a 4-pin TRRS plug is plugged into a 3-pin TRS socket.
Diagram: https://i.sstatic.net/8rSD2.jpg
"Wouldn't you need to rewire the headphones?"
This is basic physics controlling the effect here, not electrical routing. Speakers are microphones by their very design. To make them work as a microphone, you merely speak into them with them plugged into an input jack that provides at minimum a line level electrical signal to be modified by wiggling the speaker cone/diaphragm back and forth.
Yes, but the computer doesn't have the firmware to "record" that signal from the speaker output pins. Thus, to record from the speakers acting like microphones, would require rewiring the headphone cable, for the vast majority of computing devices.
If you click "record" on your computer, there's no way to tell it to record signal from the speaker output channels, even if you write a custom low-level application directly making OS calls. The OS can't even do it, because it's not supported by the firmware.
"Yes, but the computer doesn't have the firmware to "record" that signal from the speaker output pins."
No, you plug directly into the microphone jack, that is what is providing your line level reference signal that gets changed by motion in the diaphragm. Zero rewiring required.
Non-phone non-Apple devices often have a TRS microphone input separate from the TRS headphone output.
I am crap with physics but was going to say I think the last 50+ years of speaker development has been about making them less a microphone than they inherently are.
No, not really.
Dynamic loudspeakers and dynamic microphones are the same thing. They always have been the same.
They've got the knobs for the design variables turned in different directions, but they're still the same.
They even have the same frequency response whether they're being used as speakers or microphones at the moment.
Which brings up a valid way to measure the response of a microphone's design:
Use two of them. One as a speaker, and the other as a microphone. Play measurement-sounds out of one, and record the results on the other. Plot it out.
The deviations are magnified, but eliminating that magnification is just a math problem -- not an instrumentation problem. :)
They transmit sound. Anything able to detect the vibrations make it a microphone. Not sure how a speaker gets around that because it’s job is to vibrate.
The accelerometers that protect the average hard drive are easily subverted for this purpose.
There is something better. The little sensor that maintains the distance between the spinning platter and the armature is sensitive enough to be a reasonable microphone. But it is inside a heavy metal box (the HDD) so you do need to shout at it.
https://physics.aps.org/articles/v12/24
>> They tapped into the feedback system that helps control the position of the read head above the magnetic disk. When the head is buffeted by sound waves, the vibrations are reflected in the voltage signal produced by the drive’s position sensors. By reading this signal, Fu and his colleagues were able to make high-quality recordings of people speaking near the drive.
Good old video of a guy shouting in a data center https://youtu.be/tDacjrSCeq4?si=ebFDFYufOdNIU9av
The NSA could turn on your flip phones mic thirty years ago without you knowing, I don’t think they needed to do all that fancy stuff with hard drives. That’s just research that they funded to cover up the fact that they owned every computing device on the planet for a while.
I mean at this point I'm going to assume that any semiconductor device with more than a few pins has an embedded mems microphone.
I don't think that's realistic. If you're looking at the acceleration sound waves cause against a phone's accelerometer, that's likely far below the sensitivity of the sensor- phones are too massive relative to the force of sound waves from speaking. F=ma, so the acceleration you're looking at is the force of the soundwave (tiny) divided by the phone's mass (relatively large). The only reason this kind of works is because you're putting the phone on an object that's mechanically vibrating. I suppose it would work in certain situations like putting the phone on top of a large speaker, but you'd never get the resolution to decipher audio from sound waves alone for a phone sitting on a desk or in a pocket
Sounds like you've got a great idea for a proof of concept for DefCon next year...
Probably worth reviewing prior art for inspiration:
0) https://www.usenix.org/conference/usenixsecurity14/technical...
1) https://infocondb.org/con/black-hat/black-hat-europe-2014/gy...
Sounds salty. https://people.csail.mit.edu/mrub/papers/VisualMic_SIGGRAPH2... use a potato chip bag as a microphone
This is why grapheneos creates 'sensors' as a permission. On android all apps can spy on you this way.
I doubt the sampling rate is anywhere near what you would need to make out dialog in a sound recording. You might be able to tell who is speaking though if you had a voice profile.
The more reliable guitar tuners do something like this. You clip them on the neck, and they detect vibrations in the wood rather than from sound in the air.
Violin/other strings tuners tend to have this mode for orchestra uses with many people next to you also making sounds.
oddly, this is how I learned to tune my guitar. I'm pretty tone deaf, but can feel the difference between the reference string and the string I'm trying to tune through the guitar.
This is fascinating.
Thanks for sharing.
The beat frequencies are pretty noticable, just tune until they go away
That's the terminology I was looking for. Didn't know whether to tune up or down without changing frets, could tell when it was right though.
Not really. 12 TET (the tuning intervals used in almost all situations where you want equal ability to play in any key, from a Western music perspective) requires a slight amount of beating, even in those "perfect fourth" intervals between most adjacent guitar strings, and especially in the "major third" interval between one pair of adjacent guitar strings, but not in the "2 octave" interval between the lowest and highest guitar strings.
Some equal temperament intervals are narrower than their just-intonation (nonbeating) counterparts, for example an ET perfect fifth (2.996614:2) and just perfect fifth (3:2). But others are wider, for example an ET perfect fourth (4.00452:3) and just perfect fourth (4:3), and an ET major third (5.039684:4) and just major third (5:4).
If you tune each string sequentially (low E to high E, or vice versa) and eliminate all of the beating each step of the way, the effect adds up to the point where it's quite noticeable, meaning that your low E and high E will sound like garbage when played together because that ratio should be precisely 4:1 but now you've accidentally made it narrower than that. How much narrower?
For a guitar going from low E string to high E string, we need to stack 3 perfect fourths, a major third, and another perfect fourth -- and end up at 4x the starting frequency. If we use those non-integer ratios of the 12 TET system (intentional beating), we end up with 4.00452/3 * 4.00452/3 * 4.00452/3 * 5.039684/4 * 4.00452/3 = 4.00000. That's what we want. But if we use the integer ratios of just-intonation (no beating), we end up with 4/3 * 4/3 * 4/3 * 5/4 * 4/3 = 3.95062 and that's going to sound like complete ass. This is why just-intonation is not used for instruments like piano and guitar that are designed to play in all keys. It's used by choirs and barbarshop quartets without piano accompaniment, since they can adapt on the fly, and it's glorious.
An electronic tuner is the most practical way to avoid this problem. Alternatively, you could just get the perfect fourths nonbeating, get the double-octave nonbeating, and let the major third beat however it wants -- this works because rounding the ~4.005:3 perfect fourth to 4/3 is somewhat acceptable but rounding the ~5.04/4 major third to 5/3 is not.
Perfect fourth could be achieved via natural harmonics at 5th and 7th frets, double octave is just the 5th fret and voila you can get the standard tuning. Inapplicable for necks without a precise position of frets, good luck compensating height differences from nut and gauges with intonation tuners.
Oh yeah, I should've mentioned fretting the lower strings for a unison (1:1) to the higher adjacent open string. No beating in this unison yields equal temperament, due to frets scaled accordingly, so it avoids the problem I mentioned... but introduces likelihood of intonation issues (or simply bending the pitch) as you say.
I had been thinking about listening for beating in the 5:4 and 4:3 intervals using open strings, which does present the just temperament problem, and harmonics on open strings to make a unison unfortunately shares this problem 100%.
Fun idea and also I didn't know that websites could get access to my accelerometer data. However for me the sample frequency is 50 Hz which is way too low to measure even the lowest string pitch (E2, about 82 Hz).
If you know you have a single frequency close to an actual frequency of interest, you can use the fact you know you're in an aliased band to get a precise frequency estimate.
I guess thats sort of like a weird PLL thing? But I'd imagine you'd have to have prior knowledge of which string you're tuning otherwise the analysis is going to alias against every harmonic.
Every non-linear mixing of signals gives you sum and difference frequencies. It's less a weird PLL thing and more a weird trig function thing.
Its not even a non linear thing. Its a sampling thing. Even ideal sampling exhibits aliasing.
https://en.wikipedia.org/wiki/Nyquist%E2%80%93Shannon_sampli...
Yes, and. You can look at aliasing as a special case of heterodyning. Sampling is a nonlinear mixing.
Presumably there is an antialiasing low pass filter somewhere before JS gets to the data. I have a similar sample rate and it certainly didn't work at all for me.
If the accelerometer samples at 50Hz, how could there be an antialiasing filter?
What would that filter look like?
Anything physical which dampens higher frequency oscillations would act as an antialiasing filter.
What sort of size do you think something that would damp 25Hz vibrations in something that weighs a gram or two would need to be?
aka a stroboscopic measurement,
but I don't think it will work well for this case.
"the lowest string pitch (E2, about 82 Hz)."
My 6-string Kiesel Kyber bass would like a word with you while it sounds 41Hz.
The very clear and succinct description on the landing page makes me miss the bizarre antisocial charming quirk that people who made things like this used to be stuck with for their copy rather than AI generated language. Our cacophony of experience is quieting.
guitar detuner that uses accelerometer instead of microphone, it doesn't really work, but amazing to see how sensitive they are.
It also shows that it can leak all kinds of other information.
…which is why both Android & iOS put high-frequency access to the accelerometer behind an additional permission AFAIR.
It really is crazy. Placing my phone on my chest lying down i can clearly see my heartbeat on the graph
good idea https://tautme.github.io/phone-sensors/heartRate.html , try setting it on the ground and see how sensitive it is to foot steps.... or earthquake https://seismosense.com
The neat bit is that it doesn't necessarily need to sample 82 Hz directly. If the sample rate is known and the target is one of a few guitar strings, the aliased peak can still be useful. The tricky part is probably rejecting the wrong alias once the vibration signal gets messy.
Anyone got a handle on the algorithm required to do this? I've got a pocketable accelerometer-enabled device I'd like to try to implement this on..
Don't have a handle on it, unfortunately. But the algorithm is in here: https://github.com/tautme/phone-sensors/blob/main/guitar-tun.... Esp. lines 221–257 and 373–417.
The code mentions "autocorrelation" method: this is a method where you multiply the signal with delayed version of itself: result = sum(x[i] * x[i - delay] for i in some range). You vary the "delay" and pick the value that maximizes the result. This is based on the idea, that the sequential periods of the signal should be similar to each other.
Not a very good method, prone to octave errors (showing pitch one octave lower than the correct one). Furthermore, the "delay" is an integer which limits the precision, so you need to use some form of interpolation. Also it doesn't allow to recognize multiple notes sounding together. Also, slow.
You can read the paper on the "YIN" pitch estimation algorithm which describes the method in details.
I think FFT-based methods are more reliable. I did little experimentation and when measuring a pure sine wave, the frequency can be determined with high precision (tenths-hundredths of a Herz). Not so good in presence of a noise or multiple instruments - I tried to use descending from the hill optimization to figure out the pitch of each harmonic, but it didn't work out.
I implemented the McLeod NSDF pitch method, which normalizes the autocorrelation to get a pretty reliable estimate with fewer octave jumps. For precise tuning I used phase tracking between successive single-bin DFTs tuned to the target frequency.
Ah, that does look like something I can work with - thanks for the legwork, I will check it out and see if its worthwhile converting to C/C++ for my device ..
That was for a microphone based version. This is the accelerometer one https://github.com/tautme/phone-sensors/blob/main/accel-tune...
Ah, that is very intriguing indeed .. going to have some fun with this code this week .. thanks for that, kind stranger!
Can an accelerometer determine when a car is having issues? Can it do the same for a human user's body?
i remember hearing Porsche was doing some machine learning with vibration analysis to find car trouble. https://www.porsche.digital/en
Yes, predictive maintenance on motors and turbines using vibration is quite common.
Cool. It worked for E and A but it failed for string 1-4. I was surprised that it worked at all.
I mean yeah, that's cool as a fun project. And I've also heard about a project that used accelerometers as microphones for surveillance. And while it's doable, even the cheapest crappiest mic would do a much better job at recording sounds for whatever is your goal.
> even the cheapest crappiest mic would do a much better job at recording sounds
And if you don't even have that, use a speaker/headphone as the microphone, probably also better results.
It’s about what apps can do with just default permissions, no? Not about what’s theoretically possible given full access to the phone.
This sounds neat, but I think I owned a tuner for about 6 weeks before I could do it by ear... EADGBe isn't that hard.
Pack it up folks, nubinetwork has exposed the scam that is the guitar tuner industry. You don’t need a guitar tuner if you have ears; all the guitar techs and musicians who use them have bought into a lie. And obviously since guitar tuners are a waste of time, a tech demo showing that you can use the accelerometer in a commodity handheld device to pick up minute vibrations with sufficient accuracy to detect guitar tuning from a web page is just feeding into the hands of Big Tuner.
Seriously, this is the very definition of a shallow dismissal.
If you have a good external reference point. But it's also pretty easy to have your tuning drift quite a bit away from E standard if you solely rely on the strings. Getting a standard tuning is not the same as getting the standard tuning you want, exactly. This is especially true if you play in standard tunings below E, like C or B, where strings can be looser than the norm.
Pro guitar teacher here with over twenty years of experience teaching the guitar, and close to fourty years of experience playing the guitar. I still struggle with properly tuning my instrument by ear. Nothing wrong with my ears. It's just not easy to do this right.
For me it's hard because of tempered tuning, so each string should be slightly out of tune for everything to sound good. If I tuned by ear, I could get two open strings to match, but all the fretted notes and chords wouldn't sound good. On my ukulele I even tune one string down on purpose to make the fretted note sound better. And then there is the inharmonicity of overtones, and some strings have more noticeable overtones that influence how the pitch is perceived.
I can tell if my guitar's significantly out of tune, but no way I'm getting an accurate tune without a tuner.
So what do you do when the lead singer is engaging in some stage banter and you need to tune between songs? IEMs? With what reference pitch?
>IEMs
Absolutely. In orchestral band back in my high school days, between songs I would ask my friend on the tuba to give me a note for reference so I could tune the timpani. If you are in a band you should be able to manage to take a one second note and hold that by humming or something and tune up. I can't even read real music (I was a "percussionist" ie incompetent except at rhythms) and even I could do that. Better trained musicians than I could expand from there to the other notes you need.
You just need a single reference pitch. You can literally just play a sine wave into your headphones for a second on stage, or have your audio engineering guy feed you one. Or you have one of those dirt cheap tuners that clips to the fretboard to get you started.
The lead singer is engaging in stage banter partially to give you the time and space to do this. If your ensemble includes a pianist or a synth, you just have them slap a note for a reference.
A good chunk of us are in tune (pun intended) with our instruments to the point that we can simply feel the vibrations and know where we are with regards to tuning.
I haven't touched a tuner in about half a decade.
That sounds like a very useful skill for a working player. But it seems to come with a couple of significant conditions.
I can believe this is possible. But I don't think this is a reasonable thing as a baseline expectation for a player with 6 weeks of experience, which was the original comment in this thread.
I don't know the details, but I imagine you're feeling beats transmitted through the neck or something. But if that is the case (an assumption) it still requires you to have at least one known-good string, unless you're playing solo.
So, for these circumstances, and others, a pedal-based or clip-on headstock tuner seem like they still have plenty of practical application.
depends on the instrument. with acoustic instruments the resonance chamber is often tuned so that you can feel the body itself vibrating more when you're in tune
I don't doubt that this is true at some level. But is it really perceptible for 20cent differences? I don't know anything about this, but I'd be surprised. I'd say +/- 20 cents is a bare minimum for calling something "tuned", but low single digits is really the goal.
no, i still use an electronic tuner with my acoustic guitar. but i can feel the body vibrating differently when i get close to being in tune vs being a semitone off
Semitone precision is not good enough to replace a tuner. I believed that's what we were talking about. Still cool and interesting though.
i'm like this at home for sure, until i want to play along to something in tune.
i'm also the one at rehearsal literally throwing TU-3s at my bandmates who don't have tuners on their boards for some reason. you have to have a tuner if you play with others. no question.
