There’s a strange allure to single-pixel cameras due to the simultaneous simplicity and yet fascinating features that they can offer, such as no set resolution limit. That said, the typical implementations that use some kind of scanning (MEMS) mirror or similar approach suffer from various issues even when you’re photographing a perfectly stationary and static scene due to their complex mechanical nature. Yet there’s a way around this, involving a LED matrix and a single photoresistor, as covered by [Jon Bumstead] in an article with accompanying video.
As he points out, this isn’t a new concept, with research papers cited that go back many years. At the core lies the signal processing technique called compressed sensing, which is incidentally also used with computed tomography (CT) and magnetic resonance imaging (MRI) scanners. Compressed sensing enables the reconstruction of a signal from a series of samples, by using existing knowledge of the signal.
In the case of this single-pixel camera, the known information is the illumination, which is a Hadamard matrix pattern displayed on the 64 x 64 pixel LED matrix, ergo 4,096 possible patterns. A total of 4,096 samples are thus recorded, which are subsequently processed with a Matlab script. As pointed out, even 50% of the maximum possible matrices can suffice here, with appropriately chosen patterns.
While not an incredibly fast method, it is fully solid-state, can be adapted to use other wavelengths, and with some tweaking of the used components probably could cut down the sampling time required.
I would like to have seen him try illuminating 1 pixel at a time, and just assign the photodetector value to the corresponding pixel. That’s less time efficient than using the patterns, but I think it would also have worked.
Yes , it would have worked but the whole point of compress sensing is to go faster. Imagine you have a 2D ultrasound sensor, you can do a classic imaging scheme to get the result on one sensor then the other or you can find clever ways to sample the space…. Reminds me of my phd time in the 2010’s when it was the hot new thing and everyone ended up sampling in a fourrier space…
I imagine they did this to align everything. Would have been a good control to include but regardless it’s still a cool demonstration of something that’s prodominanly an academic research area
So, like the old flying spot scanners used for some early forms of television.
Of course it would have worked. But it would have taken 64 times longer: What took a few minutes here would take a few hours using that naive approach. That’s the magic of compressed sensing.
I did point by point scanning too, but didn’t show the results. It does work to create an image. Good point, it would have been nice to see it side by side. It looks very similar, but actually less full image artifacts. If there was a measurement error, it would only affect one pixel in the image. The problem was due to my homebuilt setup.
Great work! Very complete.
I believe the single white pixel you get at the top left corner is a reconstruction bias. It should dissapear if you center your measurements. Hadamard patterns form a basis only if the patterns are +-1 and not 0/1 as imposed by the optics. However, one can basically subtract to each measurement one the measurement obtained with a fully white pattern to simulate +-1 patterns.
For information, you may already know it but random patterns can work as well (if they are also centered) but then the reconstruction method must be totally different; it becomes non linear and a single sum doesn’t work. In this case, you need to solve an optimization program whose solution is the image. This is explained is certain text books and papers on compressive sensing.
Thanks for this note! I was wondering about this error. I hope I understand your point on subtracting the fully white pattern. I have this data because it is the first pattern for each dataset, so I will try it out.
I learned from another comment that I didn’t use the correct reconstruction algorithm for the random patterns. I need to learn more about this topic. Definitely disappointed about my misunderstanding because I made a point in the video about how many random patterns would be required to reconstruct an image. Do you know if the random patterns can be as efficient as Hadamard matrices in terms of SNR or some image quality metric vs. number of patterns used?
Ahh don’t be disappointed, in fact this discovery should be encouraging!
Since you got great results while doing some aspects “wrong”, this suggests you did everything else very well, and now have a clear path to improvement.
It’s also worth considering how you got such great results, whilst not perfectly replicating the “conventional” approach, as there could be previously dismissed details which your approach is leveraging unintentionally!
If you can characterize these details you could find improvements or alternative/innovative pathways to increase performance, beyond the capabilities of existing solutions!
Hadamard matrices have only +1 and -1 values. The LEDs can only provide positive light intensity, so only 1 or 0. So it is not a true Hadamard matrix. He must be using a trick.
One optimization I could think of to speed up the measurement would be to find the borders of the image first so you can use a smaller matrix. This can be done by a horizontal and a vertical scan of a line. You will get x min and max and y min and max.
Thanks for this point. Laurent Jacques’ comment was related to this topic. I just set -1 to 0 for the images displayed on the LED matrix.
Great work!
The LED matrix certainly uses PWM and there’s probably also some delay while the driving signal scans all the LEDs through the matrix. You can clearly see artifacts at 5’22 and 5’33 for example (while there’s probably also the camera’s rolling shutter involved). So i wonder if PWM + scanning delay could interfere with the measurements involved here and lower the reconstruction result quality?
Thanks! you are definitely right about the measurement errors. I had to slow down acquisition, primarily because the synchronization between the pi and arduino wasn’t great. Of course, the photoresistor was slow too.
Did not see the slow response time of photo resistors mentioned in the video or comments above. Just so [Jon Bumstead] and people trying this are aware, the response time for a photo resistor (AKA light-dependent resistor (LDR), photo conductive cell or Cadmium Sulfide (CdS) cell) can be as long as one second (https://en.wikipedia.org/wiki/Photoresistor). This burned me back in a college project years ago. And I’ve never forgotten it. Photo diodes do not suffer from this problem.
[Jon Bumstead] has a scope, so it would be easy for him to test the rise and fall response time of the photo resistor he uses. Still, it might be interesting to test if the resolution improves for slower scan times and degrades for faster scan times.
That is horrifically slow. Photodiode and their common transimpedence amplifiers aren’t light speed, but compared to >100ms they may as well be
What’s your definition of “light speed”? Even the cheap and ponderously slow jellybean BPW34 photodiode has a risetime of less than 10 nanoseconds, about the time it takes for light to traverse a small room. Photodiodes built for speed (like for fiber communications) are a thousand times faster and are still cheap.
Running this compressed-sensing rig at the speed a fast photodiode will permit (microseconds instead of minutes) will run into photon starvation issues: simple the counting statistics. That said, yes, there is ample opportunity for speed increases here, even with the LDR sensor.
It’s worth noting that the first video camera was a single-pixel device. This camera was designed to pair with the early electromechanical televisor, a primitive television system that used a single pixel for sensing and display!
Both camera and display utilized a spinning disk with holes drilled along a spiral line from the center to the edge. The holes were arranged such that only one of them would pass below a viewing window. On the other side of the disc from the viewing window, there would be a bulb whose brightness was controlled via the incoming radio signal.
That signal was generated by a similar combination of bulb using an identical disc to selectively illuminate small parts of a scene with what was called a “flying spot”. Light would reflect off the scene and be captured by a light sensor before that signal was amplified and transmitted. The biggest and most persistent technological hurdle was maintaining perfect synchronization between the disc used by the camera and the displays.
I just tried this out. Thanks again for the explanation. I have a couple examples here.
https://www.okooptics.com/hadamardbias
Some datasets have multiple white pixels along the first row. Subtracting the white frame measurement seems to always fix the first white pixel.