There's a ridiculous amount of tech in a disposable vape
blog.jgc.org729 points by abnercoimbre 3 days ago
729 points by abnercoimbre 3 days ago
For these devices the microcontroller needs to be super cheap. Microcontrollers like the Puya PY32 Series (e.g., PY32C642, PY32F002/F030) can cost in the $0.02 - $0.05 range for the kind of many-million volumes applicable for disposable vapes. These are 32-bit ARM Cortex M0 MCUs, running at a 24 MHz clock or similar, some with 24 KB of ROM and maybe 3 KB of RAM!
To put into context: this is 3x the ROM/RAM of the ZX81 home computer of the early 1980s. The ARM M0 processor does full 32-bit multiplication in hardware, versus the Z80 that doesn't even offer an 8-bit multiply instruction. If we look at some BASIC code doing soft-float computation, as was most common at the time, the execution speed is about 3 orders of magnitude faster, while the cost of the processor is 2 - 3 orders of magnitudes less. What an amazing time we live in!
Which is why when folks nowadays say "you cannot use XYZ for embedded", given what most embedded systems look like, and what many of us used to code on 8 and 16 bit home computers, I can only assert they have no idea how powerful modern embedded systems have become.
Now that it is a pity that when people talk about saving the planet everyone keeps rushing to dispoable electronics, what serves me to go by bycicle to work, be vegetarian, recicle my garbage, if everyone is dumping tablets, phones and magnificient thin laptops into the ground, and vapes of course.
> Which is why when folks nowadays say "you cannot use XYZ for embedded", given what most embedded systems look like, and what many of us used to code on 8 and 16 bit home computers, I can only assert they have no idea how powerful modern embedded systems have become.
Yet, I still need to wait about 1 second (!) after each key press when buying a parking ticket and the machine wants me to enter my license plate number. The latency is so huge I initially thought the machine was broken. I guess it’s not the chip problem but terrible programming due to developers thinking they don’t need to care about performance because their chip runs in megahertz.
There's no pressure to make a good product because nobody making this decision has to use the machine. Everywhere I've worked purchase decisions are made by somebody with no direct contact to the actual usage, maybe if you're lucky they at least asked the people who need the product what the requirements are, otherwise it's just whatever they (who don't use this product) thought would be good.
"Key presses are 15x slower than they should be" gets labelled P5 low priority bug report, whereas "New AI integration to predict lot income" is P0 must-fix because on Tuesday a sales guy told a potential customer that it'd be in the next version and apparently the lead looked interested so we're doing it.
Not just that, nobody chooses their parking spot based on the UI of the machine.
Banks and phone manufacturers now care about UI, because some of them started to do so, and people started switching to them en masse. US carriers were bleeding subscribers left and right when the iPhone was only available on AT&T, which was the first time people started switching plans to get a specific phone instead of the other way around.
People usually choose their parking based on where they want to go and how far it is from that place, and that trumps all other considerations. Paying more for programmers or parking machine processors would be a waste of money.
Interesting story; I went to park at a downtown lot in my local city (Vancouver BC) and the machine had an unusual UI. So I skipped the machine and scanned the QR code for the app. By the time I had taken the elevator up to the lobby of the building I had the app.
But then the usability on the app was so bad, that I actually could not figure out how to buy parking. The instructions were clear, but the latency on the app was unusable. The Internet connection was fine. It was the app. So I skipped the whole thing, went to dinner, and was happy when I found my car without a ticket.
"Unable to buy a ticket" would have been an interesting day in court.
> Paying more for programmers or parking machine processors would be a waste of money.
The rise of parking apps on mobile adds an interesting angle to this.
No doubt, many of us favour apps because the UX is so much better. Not quite sure if that affects the bottom line short-term, but long-term I’m sure it will.
I worked at a purchasing dept. where each commonly ordered part or service had a six digit item number that had to be entered. The CFO picked some company to do the new version of the software, and they decided to randomly assign new different item numbers which included 13 leading zeros to each item number. So now everyone had to learn the new item numbers and type in a the 13 leading zeros each time.
> There's no pressure to make a good product because nobody making this decision has to use the machine.
Most software sucks, even when people have to chose using it. Everything is buggy and slow, people are just used to software being bad.
While this is a decision-making problem, it is also an engineering incompetence problem. No matter what pointy haired boss is yelling about "priorities" ultimately software developers are the ones writing the code, and are responsible for how awful it is.
When it comes to priorities about what to write and what to focus on, the buck stops at management and leadership. When it comes to the actual quality of the software written, the buck stops at the developer. Blame can be shared.
Precisely this. We love to put our colleagues as competent victims of the system, but a competent engineer is unlikely to build an embeeded UI with high latency at their first try. It's a combination of cheap, underqualified labour and careless management.
To paraphrase Upton Sinclair: “It is difficult to get a man to prioritize something when his salary depends upon his not prioritizing it.”
Certainly one of the benefits of my "Fuck Off Fund" is that for a good many years now it has enabled me to be unburdened by concerns about whether I might get fired for saying what I think to management.
I'm at much lower risk than the imagined target of the "Fuck Off Fund" concept for things like inappropriate sexual contact or coercive control, but I find it really does lift a weight off you to know that actually I don't have to figure out whether I can say Fuck Off. The answer to that is always "Yes" which leaves only the question of whether I should say that. Sometimes I do.
And you know, on zero occasions so far have I been fired as a consequence of telling management to fuck off. But also, I had to think hard about that because, thanks to the fund, I had never worried about it. I've been fired (well, given garden leave, same thing) but I have no reason to think it's connected to telling anybody to fuck off.
This is only partially true.
If developers prioritize customer experience instead of velocity and cost in situations where that isn't warranted, the company they work for can no longer sell products as cheaply as their competitors do. This decreases their market share and their revenue, which means they'll employ fewer developers in the future.
This is almost an evolutionary process, many (but not all) markets choose for developers which don't care about such things.
> When it comes to the actual quality of the software written, the buck stops at the developer. Blame can be shared.
No. The quality is not prioritized by management. A dev that fails to ship a feature because they were trying to improve "quality" gets fired.
We have no labor power because morons spent the good times insisting that we don't need a professional organization to solve the obvious collective action problem.
The idea that workers are not responsible for their own competence or the quality of their work output is such a bizarre take that you really only see on HN. Just because nobody is forcing you to write quality code, doesn't mean you shouldn't. Nobody is forcing you to bathe or brush your teeth, either, so why do we do it?
My first guess was debouncing. They assume that the switches are worn out, deeply weathered, and cheaply made. Each press will cause the signal to oscillate and they're taking their sweet time to register it.
When the device is new this is an absurd amount of time to wait. As the device degrades over 10, 20 years, that programming will keep it working the same. Awful the entire time, yes, but the same as the day it was new.
I was late for a train at my local station and the parking machine was taking ages to respond to keypresses. I could see the training pulling up to the platform and I was still stuck entering the second digit of seven. In my shameful frustration I hit the machine fairly hard. While the button presses might take a while to register, the anti-tamper alarm has really low latency and is also quite loud.
You need to find the right person to complain to. Here we are sympathetic, but can't do anything.
The right person is the other riders on the train - but the hard part is to frame this such that they join you on a march to the the agency that owns that machines to complain. I wish you the best of luck figuring out how to do that (I don't know how to do it - and if I did there are might higher priority things that need to be fixed).
Debouncing would be smart, sure. But sometimes, these sorts of embedded machines are weirder than that.
At Kroger-brand gas stations near me, I get to interact with the buttons on gas pumps to select options and enter a loyalty ID.
Those buttons have visible feedback on a screen, and also audible feedback consisting of a loud beep. And there's always delays between button press and feedback.
Some combination of debounce and wear might explain that easily enough.
Except... the delay between pushing a button and getting feedback is variable by seemingly-random amounts. The delay also consistently increases if a person on the other side of the pump island is also pushing buttons to do their own thing.
It's maddening. Push button, wait indeterminate time for beep, and repeat for something like 12 or 13 button presses -- and wait longer if someone else is also using the machine.
I can't rationally explain any of that variability with debounce.
They are running it on a Java VM in a container - on a 386?
Over WiFi?
Perhaps.
Or perhaps the original programmers skipped the class on concurrency 25 years ago, and nobody has subsequently bothered to pay anyone to update that part of the software.
One time I decided to test whether these grocery story loyalty card XX cents off per gallon transactions were properly isolated, when my wife and I were both filling up vehicles at the same gas station at the same time. We both got the $0.50 discount per gallon with no problem. I'm sure there are lots of creative ways you can exploit the poor design of these things.
That's a good point. When I use them I assume they're making API calls to a central server to validate (or something) them.
Making API calls to a server to do button debouncing does sound like something so stupid a tech company would do it
in the worst case, they don't know they're doing it, because they've called some 'debounce.js' microservice wrapper and haven't audited it
One of the more inspired design choices of the parking ticket devices in my area is the inclusion of a key repeat feature.
If you keep your finger on the touchscreen for just long enough, it helpfully repeats the keystroke while you're entering a license plate.
Given the inevitable hardware issues, this means that what should be a single tap frequently becomes a burst of identical characters.
The programmers who worked on this probably would've liked to be game developers instead.
That's programmer incompetence. Unfortunately pervasive, especially with devices like parking meters, EV chargers, and similar, where the feedback loop (angry customer) is long (angry customers resulting in revenue decrease) or non-existent.
It could be a management problem instead also, while developers are just following instructions sent by management
They also prefer you to use the mobile app so they can gather more data so they do not want the devices to work well in the first place.
It's a nice theory, but many of those terrible parking ticket machines predate smartphones, so it might be the case for machines built now, but it's really hard to imagine that that was the original intention
I work in an adjacent industry, and trust me when I say that a lot of older equipment companies just did not care much about the experience of using the equipment. It's much more important to tick all of the boxes in the back end accounting system than to have a high quality experience on the kiosk.
It's organizational incompetence driven by companies that see software development as a cost centre rather than a key asset.
It's usually clear when this has happened. Buggy bargain basement output.
Give it some slack, it's probably doing its best to inexplicably run windows.
Disagree. Windows for embedded runs extremely well, though can take a minute to boot.
My underpowered cash register that hadn't been updated in a decade could run POS on top of Windows 7 Embedded POSReady buttery smooth.
Occasionally they would start performing poorly, and it was always a network issue.
Or maybe they think they should be sending each keystroke to a server and waiting for the response.
A server on Mars?
Na each key press goes to a separate lambda invocation that gets submitted to a Kafka queue, and what happens after that is a mystery to all involved.
We can make crazy latency ourselves just fine, no space transmission necessary
No, not a mystery, in fact.
Each keypress is appended to an 80 line prompt (key name along with timestamp of keypress and current text shown on the screen) and fed to a frontier LLM. Some of the office staff banged on the keypad for a few hours to generate training data to fine-tune the LLM on the task of denouncing key presses.
Thanks to some optimizations with Triton and running multi-GPU instances, latency is down to just a few seconds per digit entered.
You see, we needed to hit our genAI onboarding KPIs this quarter…
Probably a Celeron-powered PC tower barely keeping up with Windows Server 2008 R2 in a closet of a public office ;)
The server is probably running Python
lol it's the flask debug server, "don't use this in production" banner and all
Everyone was locked out in a building am staying at (40 something stories) for several hours. When I asked the concierge if I can have a look at the system, it turns out they had none. The whole thing communicated with AWS for some subscription SaaS that provided them with a front-end to register/block cards. And every tap anywhere (elevators/doors/locks) in the building communicated back with this system hosted on AWS. Absolute nightmare.
I wonder what happened to the building when us-east-1 went down.
Now I am waiting for time when they move us-east-1 physical security to run in us-east-1... Thus locking themselves out when needing some physical intervention on servers to get backup.
Facebook already got bit by this when their BGP setup pooped its pants on Oct 4, 2021
They could ask facebook for advice.
https://skeptics.stackexchange.com/questions/52448/were-face...
I wonder what happened to the building when the internet went down. How do you get into the room to reboot the router?
There is probably a break-glass procedure for such cases, like, break the literal window.
> Absolute nightmare.
Yes, but still probably a million times easier for both the building management and the software vendor to have a SaaS for that, than having to buy hardware to put somewhere in the building (with redundant power, cooling, etc.), and have someone deploy, install, manage, update, etc. all of that.
>than having to buy hardware to put somewhere in the building (with redundant power, cooling, etc.), and have someone deploy, install, manage, update, etc. all of that.
You don't need any of that. You need one more box in the electrical closet and one password protected wifi for all the crap in the building (the actual door locks and the like) to connect to.
And when that box fails, you're looking at how long with no access? Longer than any AWS outage.
The IT guy walks in and replaces/restarts the box instead of waiting for the gods of AWS to descent to earth and restart theirs. They have direct control vs. waiting for something magic to happen.
You also have real-time ETAs from an actual human local to the issue. Plenty of domains where your clients won't care if AWS is down for everyone.
The building has an onsite IT guy with enough spares to fix anything that could go wrong with the box?