Show HN: DIY virtual HDMI monitor using "AR" glasses
github.com112 points by blensor 6 days ago
112 points by blensor 6 days ago
I am making a virtual HDMI monitor using Viture Pro XR glasses and an SBC ( currently OrangePi 5 Plus because it has HDMI-in ).
What it does is map the frames from the HDMI input onto a virtual display that is controlled by the IMU data from the glasses ( 3DOF only ). I've put AR in quotes in the title because many won't view those display glasses as true AR but by tracking the head movement it comes close.
I am trying to build kind of a "low cost" version of a virtual screen that acts like a monitor and can be connected to anything that has an HDMI output
I started off using the official Viture SDK to interact with the glasses but have since switched to a reverse engineered implementation of the protocol because their SDK is not available for ARM
Here is a video showing the first version: https://www.youtube.com/watch?v=D6w5kAA22Ts
Big caveat: Performance still needs to improve a lot because the whole frame reading/converting is completely unoptimized for now.
What other solutions do exist out there? * Streaming the computer screen to a headset like Meta Quest/Vision Pro * Connecting a HDMI capture dongle to the Meta Quest directly * XReal Beam ( basically the same as this project but official and for XReal glasses )
And for the obvious question, why I am not use something like a Quest or Vision Pro 1. Comfort 2. Price 3. Comfort
After using those display glasses over HMDs it's hard to convince myself to use a headset for productivity again
I think what is not clear from your project description is that your AR glasses don't provide an environment locked virtual screen, meaning a screen that stays fixed in space while your head moves. The native screen of your glasses provide a screen that is fixed relative to the glasses, like having a display locked to your face. Some AR glasses do provide this feature, so it overcomes a product flaw in AR glasses and will at some point not be necessary. Your project could also allow for other features that the AR glass manufacturer didn't think of, or has gated behind upgrades and product segmentation. Is my assessment correct? I use AR glasses heavily for work and other things. I'm actually typing this on a pair right now. I've never understood what the application for environment locked screens is other than novelty/marketing. My glasses provide enough sensor data to implement this but I just can't be bothered. I guess it makes a lot more sense if you're emulating more than one monitor, without environment locking you'd only ever be able to see the sides of the monitor(s) beside the one you're looking at. But yeh for a single monitor, I guess it takes a bit of getting used to but non-locked seems far more reliable I'm curious how/what you use the screens via AR for if you're not using environment locked screens. Particularly in a productivity/work environment. Unless I'm misunderstanding the feature, it seems like enironment locked screens allows for more natural usage and interactions with the screens in the virtual space? My experience is mostly with VR/AR products like Oculus has been mostly with environment locked AR information. I suppose they could? I prefer having my posture decoupled from what I'm looking at though. It's like having a very nice monitor that uses ~1 watt of power and happens to be positioned exactly wherever is most comfortable without even having to think about it. It's way better than a normal monitor if you don't have to do eg pair programming. How are you finding the focus? I use the Xreal Air 2, but the edges are blurry, and I can't get the glasses close enough to my face to see the entire screen in focus, even if the top of the glasses is touching my forehead. That's interesting. I think mine are the original Xreal glasses? I had to slightly adjust the nose part but otherwise the focus has been great for me. That would be a shame if their regressing. I think the use-case for these is more VR focused, with the AR just being a "being able to notice when something needs your attention" feature (where you would respond to such an interrupt by taking the glasses off, not by trying to look at the interrupting thing through the glasses.) I've heard people propose that these "screen in glasses" devices (like the Xreal Air) are useful for situations where you want a lot of visual real-estate but don't have the physical room for it — like in a dorm room, or on a plane. (Or at a library/coffee shop if you're not afraid of looking weird.) --- Tangent: this use-case could likely just as well be solved today with zero-passthrough pure-VR glasses, with a small, low-quality outward-facing camera+microphone on the front, connected only to an internal background AI model running on its own core, that monitors your surroundings in order to nudge you within the VR view if something "interesting" happens in the real world. That'd be both a fair bit simpler/cheaper to implement than camera-based synced-reality AR, and higher-fidelity for the screen than passthrough-based AR. † Which wouldn't even need to be a novel model — you could use the same one that cloud-recording security cameras use in the cloud to decide which footage is interesting enough to clip/preserve/remote-notify you about as an "event". The most obvious advantage is being able to "zoom in" on the screen by moving closer to it (as with a real monitor), which is impossible with 3DOF or view-locked XR. Yes that's correct, and it could be expanded to other features yes, one thing I've been mulling over is mult monitor input which you don't even get on devices that support what I did here natively It is indeed hard to assess from the video at 19s? (Relatively impressive thru-the-glass perspective ntheless!) https://www.youtube.com/watch?v=D6w5kAA22TsP&t=19s Is it me or is there some dynamic here that could/should be further exploited to implement what you suggest It's also a Viture-specific project, but I suspect if the author wanted, this could expand to other video display glasses in the future and he could simply rename the project. Spot on. And yes expanding it to other glasses ( which are popping up quite a lot recently ) would for sure be possible I'm actually wearing the Xreal Air 2 right now, and yes, this is basically exactly what their app does. Personally, I only need one screen, so I'm using the glasses without the app, but this is great if you want a virtual second display somewhere in the room! Very well done. I’m staying at a friends and using my laptop and a second monitor of his to work. I’ve wondered if the Xreal Air 2 would be better than a large monitor. My hunch is that having a dedicated smaller monitor not on my face is more comfortable and productive than a large virtual one. From your experience would you agree? Hm, no, I much prefer the Xreal to the laptop's screen (except for the blurry edges). I don't care about the monitor size, only the resolution, but I'd rather have the monitor always be straight ahead in my vision (no matter how my head is tilted), and I really like the Xreal for that. As I said, though, I always only use a single monitor anyway, so YMMV. I haven't previously heard of Xreal. Looking at their glasses, it seems like they're all 1080p. I find it very difficult to deal with text on a regular 1080p monitor after having uses 4k monitors for a while. Is there something about the way the glasses work that makes 1080p glasses equivalent/better than at least a 1440p monitor in terms of clarity? I have them and they actually look worse than a real 1080P screen because the edges are out of focus. I know some people do, but I could never use them for real work. They’re ok for videos and video games but not amazing. If they were 4K they’d be significantly more usable but for me the number 1 issue is the edge blur and general clarity issues which I guess are lens related. Well, 1080p is 1080p, what counts is density. The display looks huge on the glasses, so density isn't great, but it doesn't bother me personally. YMMV, though. The Xreal Air 2 Ultra look pretty interesting. After using both 3DoF and 6DoF for years, I just can't imagine using a 3DoF headset for more than watching videos or simple writing tasks. I'm actually a bit jealous about the XReals. Back when I bought the Viture I wanted to buy XReal first but they had such strange shipping policies ( they shipped to every county around me but not to Austria ) so I went for the Viture. The Xreals are the only ones I've tried, so I don't know how they compare, but I have trouble with fit. I have trouble in general, because of my nose bridge being too wide, but even when I put the glasses right against my forehead, I still can't get the entire screen in focus, the edges are blurry. I really wish the screen were smaller and more to the center, then they'd be perfect. I guess it depends on whether you want them for games (the important stuff is usually in the centrer) or for work (the important stuff is usually at the edges). I really like the focal distance, though, it's around 5m away and pretty comfortable for the eyes. With the viture I have the problem that the bottom of the screen is almost always not visible. I initially used the accessibility feature that let's me zoom into the desktop with mouse follow moving the viewport but that gets old quickly, which is one reason why I started that project. I'm also planning to add other "display" modes like a curved monitor and in theory we could connect multiple monitors using several hdmi capture devices but that's probably even less useful I also have this problem with the Viture glasses. Do you know if this accessibility feature is available when using it with phones? I haven't found a way to do it on the phone unfortunately which is really annoying. You could in theory use the approach I posted here and use a USB to HDMI adapter to connect the phone to device and it would do the same thing it does with a regular PC but that loses all the benefits of being small and portable if you have to carry around all that additional stuff If you have 3dof tracking only, consider making the displayed image to have no stereodisparity, this way it is optically correct to being a screen projected onto the celestial sphere. This is also how reflex sights works on fighter jets and guns. I have to look into that, thanks. I've been thinking about combining it with our work on the reverse engineered HTC Vive Ultimate Tracker stuff here https://youtube.com/shorts/U4vhEPQw-Uo to get 6DOF But realistically I ( and nobody else ) won't strap the tracker to my head all the time https://immersed.com is a great piece of software that gives you multiple virtual screens. It supports all major OS and VR devices, with their own https://visor.com headset coming soon (tm) Just venting and ranting about AR glasses here: One thing about AR glasses that I don't get so far: Why do ALL of the AR glasses use only 1080p displays per eye whereas all the other VR solutions that also have a display right in front of your eye have even bigger resolutions? I really do want AR glasses that can act as an alternative to a physical display. But they're always 1080p for some reason. The companies I've seen making them appear to me to keep fixing important issues and adding really useful features - but seem to avoid touching anything that would improve the resolution. Edit : There's a part of me that believes the situation here is a lot like how Linus Tech Tips described the Dashcam situation. The display panels used for AR(and some pre-Oculus VR) displays are completely different thing from normal LCDs. Lots of them used to be a reflective panel made of silicon wafers with RGB front lights flashing in sequence. Driving circuits and interfacing protocols are all non-standard, under NDA, and quite complicated. The panels also needs to be coupled to a complicated optical lens - they're just magnifying glasses, only fancier. The complete thing that has a digital input and an optical output is often referred to as an Optical Engine. What that means is, these glasses are made by someone paying $$$ once to Asian engineering companies to have it figured out, and everyone reuse that exact setup for years on. And companies like Sony or Canon, they develop stuffs by scraping engineers off a wall instead of first throwing them at the wall, so every Sony or Canon cameras and projectors come with novel viewfinders and crazy patented lenses. Or panel suppliers like Kopin, Epson, or Sharp can arrange contacts to engineering consultants. I guess. Same likely goes for all the Chinese companies too, though I'm not familiar enough on that front to be able to offer googlable keywords. Startups and even mid-sized consumer electronics companies don't have that kind of time or financial backings or margins to do the same. They barely manage to pay for assembly and ship it. And so the spec of the final product is whatever spec of parts that they could buy off the shelf. What about Apple? Well, they pay to have 2 extra display factories built for redundancy by policy - I'm sure that most military organization don't do that. And even then they use a rather simplistic, completely concentric and rotationally symmetric optical design. (I kind of have a crazy idea to bypass some of that, but unfortunately I'm crazy and so is the idea) now I'm intrigued, any hints? Straight up homebrewing display devices. How hard can it be. It's depressing but not surprising when you remember that that only reason any of this exists is because Lucky Palmer pointed out that you can kickstart modern VR R&D with off-the-shelf smartphone displays (IIRC). In my experience you need at least 8k per eye to get sharp text. But if you just want to watch a movie 4k will look very crisp and 1080p@30hz also works for movies. So what you get is basically a larger mobile screen for watching videos, something that competes with video projectors, but you can use it on the bus. It's an easier sell compared to a device that will give you a good reading experience and competes with paper books. Those of us that want to use them as terminal emulators are still too few. Not disagreeing with the resolution but I assume for most last-gen use cases it didn't make much sense because you also need to be able to drive that somehow and you can be happy that you get a 2 Lane display port interface up until the more recent phones. So if you have high res displays that no one can realistically use for the intended mobile cinema use case then you are wasting money and energy The screens are two entirely different products. The current Sony micro-OLED screens being used by product manufacturers for AR glasses require out-of-plane reprojection versus VR headset screens which require focal lenses. The top-of-the-line solutions by Sony used in current products only go up to 1080p120. I heard something like this in the Linus video. That video also mentioned Sony as well. I don't know what I would do with higher resolution personally. This is already enough that xterm's "tiny" font is extremely hard to read. Personally I'd prefer lower power draw to more resolution. AR glasses have smaller FOV which means that a lower resolution can still achieve a higher peak pixels per degree compared to a VR headset. One of the reason is IMHO that the optical resolution with those lense designs does not get better than 1080p. It seems even that resolution is blurry at the edges. So anything beyond probably would be a waste without totally different optics. This feels like the most probable reason. Likely a big redesign would be needed for larger resolutions. It's a packaging problem. They don't have the luxury of placing relatively large screens in front of the eyes. AR solutions value mobility over fidelity. This has numerous repercussions, leading to those displays. Nice! I wish this worked on my XReal Air :( It does have IMU sensors because their own app (nebula) uses them, and even does 3D effects. By the way I find my Quest 3 way more comfortable than the Xreal. The problem with the xreal is, the prescription inlays are too small so I can see only a part of the display sharp unless I put in lenses every time. And it is kinda heavy on my nose and ears. the Quest is super nice, has a wider FoV. But it's heavier and bulkier to bring along of course. PS I had no idea you could connect an HDMI dongle to the quest. I'll check that out. If I had XReal I could look into that too, it's probably pretty similar to how it works for the Viture. Regarding the HDMI on Quest 3, I believe it was sometime last year when they made that possible https://www.meta.com/help/quest/1350679463006443 Yeah I bet it's real similar. But I understand. Thanks for the heads up about that Quest thing, I actually have some of those adapters so I will give it a try. Wow, $409 getting you a AR display that's viable to hack on is crazy. I can't believe how approachable hardware is for tinkerers some time. Cool project, but I'm missing what does this "remapping" do that the glasses can't do by themselves? I see the glasses are connected (via usb-c?) to the OrangePi. Is this project converting HDMI signal to the Viture protocol? if so, can't you run a screen grabber directly on your machine and send it to the glasses? If you connect the glasses to your computer you get a fixed screen in front of your eyes (like those display glasses back in the day that promised to give you a huge screen), which is fine for video viewing but if you want to use it for productivity you want to be able to look at certain parts of your screen vs. straining your eyes to glimpse the edges. That's where this comes in, it allows you to look around the virtual screen with the added bonus that you can connect it to almost anything and behave the same not just your PC > What other solutions do exist out there? Xreal One and One Pro have this built into the glasses themselves. The Xreal Eye camera is meant to add on-glassed 6DOF tracking as well. I looked at those Viture glasses, but the pitch of the glasses (how far the eyes are apart) looks strangely low. Perhaps this works in the Asian market, but it just seems inadequate for the Western world. I own them, bought just recently. The eyes have these circular wheels near your eyebrow above the frame that lets you independently adjust the distance for each eye, if that's what you're talking about. Good to know, but does that move the arms too? Some people have bigger heads than others. It does not, but for what it's worth, I have been told I have quite a large head and they fit comfortably. Ok, one more question: does the wheel also make the black glasses move? Because I'd imagine that if only the internal optics move to the sides, you get to have less protection from external light on the sides of your view. I think they suit bigger heads as well is because the hinge on the stems can go slightly beyond 90° Did you fix the op5plus HDMI in driver? Thing was unstable like 8-12 months ago So far I haven't had any issue but I haven't done long tests yet. I'm using BredOS not their official OrangePI images though I, for one, would like to see AR glasses that use e-ink. Black and white with a slow refresh rate. Would be perfect for my use-cases. This exists, sort of. I got a demo of these at a YC event a couple months ago and was very impressed. The back is opaque, though, so it's not real AR, but it does accomplish "E-Ink displays in a glasses form factor": It's something... not what I would like personally, because it's literally only a kindle in your eyes. I'm looking for a "here's a bit of e-ink in the corner of your lens" kind of thing, with a refresh rate of say 1 fps you still would need a backlight but it's an interesting idea You can't backlight an eink display. They are opaque, by design. Thats why they make such great ebook displays, so you can take them outside and still read them. Eink displays require .. i guess 'forelight'. Light that reflects off the display front. I didn't know that, I just wonder how the kindle with light works though.
Is that light coming from the front too? If you look closely at a backlit kindle screen, you can see that the e-ink pixels are a decent distance behind the surface. It’s more noticeable if there are scratches or dust on the screen and you rotate the display. The surface reflects the light from the sides back to the e-ink. IIRC, side illuminated microscopic dots on a protective cover. Light only escapes through the dots, illuminating the panel, without shining into the user. Would be cool to wear a "light panel" on your sleeve. Outdoors you see the e-ink fine. Indoors you look down at your illuminated sleeve to see the e-ink in your glasses.
sitkack - 6 days ago
msgodel - 6 days ago
bodge5000 - 6 days ago
veilrap - 6 days ago
msgodel - 6 days ago
stavros - 6 days ago
msgodel - 6 days ago
derefr - 6 days ago
underlipton - 6 days ago
blensor - 6 days ago
gsf_emergency - 6 days ago
andrewmcwatters - 6 days ago
blensor - 6 days ago
stavros - 6 days ago
mentos - 6 days ago
stavros - 6 days ago
0xffff2 - 5 days ago
einsteinx2 - 3 days ago
stavros - 5 days ago
k__ - 6 days ago
blensor - 6 days ago
stavros - 6 days ago
blensor - 6 days ago
gavinray - 6 days ago
blensor - 6 days ago
xeonmc - 6 days ago
blensor - 6 days ago
verdverm - 6 days ago
LonelyWolfe - 6 days ago
numpad0 - 6 days ago
blensor - 6 days ago
numpad0 - 6 days ago
underlipton - 6 days ago
z3t4 - 6 days ago
blensor - 6 days ago
andrewmcwatters - 6 days ago
LonelyWolfe - 6 days ago
msgodel - 6 days ago
charcircuit - 6 days ago
riedel - 6 days ago
LonelyWolfe - 6 days ago
kevin_thibedeau - 6 days ago
spookie - 6 days ago
wkat4242 - 6 days ago
blensor - 6 days ago
wkat4242 - 6 days ago
skeptrune - 6 days ago
deivid - 6 days ago
blensor - 6 days ago
Aaron2222 - 4 days ago
amelius - 6 days ago
gavinray - 6 days ago
amelius - 5 days ago
gavinray - 5 days ago
amelius - 5 days ago
blensor - 5 days ago
benhoff - 6 days ago
blensor - 5 days ago
unsupp0rted - 6 days ago
alex1115alex - 3 days ago
unsupp0rted - a day ago
blensor - 6 days ago
sleepybrett - 6 days ago
blensor - 6 days ago
ashdksnndck - 6 days ago
numpad0 - 6 days ago
unsupp0rted - 6 days ago