This is really cool and very clever. But i want to raise one thing.
> designed a special color reference chart that can be printed on a card
My rudimentary understanding of physics makes me suspect this sentence is a simplification.
A normal printer use Cyan Magenta Yellow Black to print. A photo of such a print would already destroy alot of spectral information for the same reason the individual rgb sensors do.
So i suspect those colored dots are a very careful and deliberate concoction of very particular inks with very specific spectral color bands.
I suspect alot of effort went into finding, mixing and algoritmically combining the right inks.
I'm guessing it works similarly to a how a narrow band florescent lamp makes only materials that reflect a very specific frequency be visible, which makea alot of prints and pigments look wierd. (If you do the opposite; use ink with very specific spectral band, you can instead measure the lamp)
Insanely clever. (Whatever they did)
kelseyfrog1 day ago
Not just that, but it would presumably be sensitive to light emission spectra too. As inks can only reflect wavelengths of light that hit them, if the emission spectra has spikes or gaps - think LED or florescent - the reflected spectra will be a function of the light source[1].
Perhaps there's some accounting for this, and I'm curious to learn what it is, because it's a phenomenally complex problem.
1. You might think the sun is a standard source, but it's usually modulated by the atmosphere[2].
2. Unless you are in space.
aDyslecticCrow1 day ago
> Perhaps there's some accounting for this, and I'm curious to learn what it is
The slip itself is a calibration reference, so a clean photo of it could serve to compensate for the lamp and camera and calculate how accurate the readings is for different parts of the spectrum. (But good wide spectrum light would be ideal for high precision readout)
You're also still limited to visible light because of the camera uv and ir filter, for which the sun is a decent reference.
kelseyfrog1 day ago
oh, yes of course! Thank you :)
voidUpdate18 hours ago
I don't think the sun is even a perfect source when you're in space, doesn't it have gaps in its emission spectra from the gasses that make it up?
actionfromafar16 hours ago
You are quite right, I had no idea.
"Between around 10,000 nm (far infrared) and around 100 nm (deep ultraviolet), the spectrum of the Sun's spectral irradiance agrees reasonably well (though not perfectly) with that of a blackbody radiator at about 5,700K. That is about the temperature of the Sun's photosphere. The deviation from a perfect blackbody spectrum is due to many factors, including the absorption of light by constituents of the solar atmosphere, and the fact that the photosphere is not uniform, but has some hotter and some cooler regions, so that what is seen from the Earth is a composite spectrum of blackbody radiators at a range of different temperatures. About 99% of the total electromagnetic radiation coming from the Sun is in the ultraviolet-visible-infrared region."
I was initially surprised too, but I think there is a some trick...
I'm not sure what happens on paper, but when you have ink disolved in water the abortion is not linearly proportional to the concentration, it's exponential. For example, consider a red ink and 5 magical selected frequencies and the absortions at 1% of concentration are: 99%, 99%, 50%, 10%, 1%
If you double the ink at 2% you get 99.99%, 99.99%, 75%, 19%, 1.99%
So increasing or decreasing the ink concentration may give you information of different frequencies, even with only one ink and only one sensor. In this case mostly about the 3rd and 4th. With more concentration you may kill all the light in the 3rd and measure the absortions ratio between the 4th and the 5th.
One problem I see here is how to order them, but I guess it's possible with a few sensors and a few inks. Each sensor sees all the frequencies, but with different weight. I'm not sure if it's possible to solve this, but perhaps you need some initial approximated model of the inks(???).
Now, when you put the inks on paper and have a unreliable light source and perhaps other technical problems, ...
In conclusion, I think it's possible to use different saturation and mixes of the inks to get different spectral distribution of the light that bounce on the card. Then use the three sensors to get three averages and try to use big linear algebra book to reconstruct what happens in between. But I should read the paper to be sure.
Karliss1 day ago
Based on supplementary material "We utilized a professional photographic inkjet printer
(ImagePROGRAF PRO-1000, Canon), equipped with 11 ink
cartridges and a chroma optimizer". Not your typical office printer but nowhere near as exotic as you might expect.
lupire1 day ago
How special are the inks in the cartridge, though?
Almondsetat9 hours ago
They are special in the sense that they are calibrated and QA'd for professional use
actionfromafar16 hours ago
I have to assume it used inks available from Canon. Making your own inks is a research project on its own, so my guess is that would have been mentioned.
chankstein381 day ago
Wouldn't it be nice if they just told us so we didn't have to speculate? This is cool stuff and I'm glad I know about it but, as someone interested in this field of study, I'd love to try this out. But I guess I should stop being surprised when even a company like IEEE can't be bothered to write an article with any actual information. Just a bunch of simplified summarized crap.
tzs1 day ago
At the bottom of the article is a link to the paper, which is open access.
aDyslecticCrow19 hours ago
I missed that! Thank you!
greggsy12 hours ago
I would assume that with time, you could just print it on a generic $30 inkjet, and calibrate the card, sensor, and whitelte balance to a stored reference image.
It won't be as accurate, but it might be enough to offer some insights into whether liquid photographed in the article is in fact whisky, not urine (which to me seems to be a much more noble demonstration subject).
privatelypublic1 day ago
Ink is perfectly capable of being a phosphor, in which case it'll up or down convert wavelength X to wavelength Y.
aDyslecticCrow1 day ago
My gut feeling is that finding enough very specific wavelength shifting inks would be harder. Perhaps its a mix though to get good readings in the faar edges between the rgb wavelengths.
I hope there is a research paper on this i can read.
privatelypublic13 hours ago
I expect rolling shutters to be a far larger issue. Haven't read the article a second time, but non-uniform and inconsistent lighting is a huge challenge for this kind of work.
We utilized a professional photographic inkjet printer
(ImagePROGRAF PRO-1000, Canon), equipped with 11 ink
cartridges and a chroma optimizer (PFI-1000 LUCIA PRO Ink,
Canon), and used the manufacturer-recommended genuine
paper (Photo Paper Premium Fine Art Smooth, Canon) for
printing. To reproduce the desired reference colors for the
spectral color chart, we also implemented a customized printing
calibration process while maintaining the International Color
Consortium (ICC) profile. The actual printed colors (output)
showed notable distortions compared with the intended colors
(input), which were particularly influenced by the type of paper
(print sheet). For customized printing calibration, we mapped
the exact relationship of the CIE xy chromaticity values
between the digital color input and printed output values. After
the printing process was completed, we measured the
reflectance spectra of all reference colors from the printed
spectral color chart (Fig. S1) using a spectrometer and a diffuse
reflectance standard (equivalent to using CIE illuminant E). We
confirmed that the CIE xy chromaticity values obtained from
these measurements were in excellent agreement with the
desired input values within the SWOP v2 gamut (Fig. 1(e)).
RicoElectrico1 day ago
Printing can use so-called spot colors.
dylan6041 day ago
I similarly thought that just because they said print does not mean it was printed on someone's ink jet. I'd hate to see how many different Pantone colors might be necessary.
altairprime1 day ago
If you only need one card per 10,000 photos, then the cost of the card starts to look cheap compared to a spectrometer and its bulk.
mvhv1 day ago
This doesn't really seem like "hyperspectral imaging". I think the idea is having a reference colour chart of known emission characteristics and photographing it through a transparent substance gives you an idea of how much that substance attenuates each wavelength.
It's a cool trick if it works, but it seems very finicky and I guess would be limited to transparent/homogeneous liquids?
atwrk15 hours ago
On top of that it only works in the VIS range, thanks to the filters in front of the camera sensors - and most of the interesting information is in UV or IR. VIS only contains information about a few elements. (see Fraunhofer lines https://en.wikipedia.org/wiki/Fraunhofer_lines)
crazygringo1 day ago
In theory maybe you could build a version made of inks printed on a reflective mirror? And then you would hold the mirror so it reflected the object into the camera?
But that seems far more difficult. Precisely combining and applying combinations of inks to a mirrored surface sounds like a helluva manufacturing challenge.
mvhv16 hours ago
I think ink and mirrors are kind of fundamentally incompatible.
Probably closer to what you're thinking about would be putting a bunch of tiny bandpass filters infront of a mirror, but in that case you can ditch the mirror entirely and just point the camera through the filter array.
A filter array right on-top of the sensor is how (the vast majority) of) CMOS cameras distinguish colour anyway.
fkyoureadthedoc1 day ago
> The new patent-pending technique
> “Every photo carries hidden spectral information waiting to be uncovered. By extracting it, we can turn everyday photography into science.”
And with our patent, extract rent from anyone who wants to do it!
moritonal1 day ago
That's a bit of a bad faith take. You were welcome to go spend the years(?) this chaps dedicated to putting together the research required to build this. If it works, let him enjoy the fruits of labour.
dtj11231 day ago
Sure, if he'd come up with this primarly using his own resources and time, but he discovered this whilst being paid to conduct research at a public university, a form of institution which is explicitly intended to disseminate knowledge. Society should enjoy the fruits of its investment.
brookst14 hours ago
Would the university have been willing to invest in the exotic printer and the labor to do the work without the potential upside of a patent?
It’s very easy to declare that someone else “should” do a bunch of work and spend a bunch of money for the altruistic benefit of society.
SequoiaHope1 day ago
Patents do more than let you enjoy the fruits of your labor - the market already allows for that. Patents use the force of law to bar anyone else who might have discovered the same thing from building upon it.
spookie1 day ago
Imagine you are just a dude, you did all this work, and go to "market".
You are just a dude, therefore business grows slowly.
You gather enough attention that some corporation with a lot of bling just goes and copies your thing.
Your business fails.
SequoiaHope1 day ago
Once can just as trivially construct an argument demonstrating the issue with patents but the problem with this style of argument is that patents are not a simple thing. They have global far reaching effects. The government distributing a monopoly on information is a serious interference with the market, and due to patent harmonization efforts across the world, one person filing a patent in New Jersey affects even people in Kenya and Turkey and Thailand. The arguments for patents are often, as I see it, based on a deeply flawed understanding of the motivations of innovators and the affects of open information on innovation. For example most arguments in favor of patents cannot explain how open source works, and so are clearly incomplete or outright wrong.
fluoridation21 hours ago
>For example most arguments in favor of patents cannot explain how open source works, and so are clearly incomplete or outright wrong.
Can you clarify this? Just curious about what you mean.
SequoiaHope4 hours ago
I often speak to people who say that no one would innovate without government-secured monopoly on information, and this is clearly false as open source works without this monopoly. As such it seems to me that people who make that claim don’t understand how innovation works.
Often I say that if we phased out patents and other IP restrictions, then investments would not stop but they would change from less frequent large investments to more frequent small investments. As long as designs can stay secret until release, there will always be first mover advantage and brand recognition. But you might get smaller investments to build out manufacturing for the next year, rather than bigger longer term investments. The flip side is that stagnant innovators who got lucky once will be subsumed by more agile competitors who can better deliver those innovations to market.
Thus investment and innovation wouldn’t stop - markets still ensure certain advantages for innovators - but the nature of investment and innovation wouldn’t shift towards more incremental moves and more diverse actors. A major upside to this is that those best suited to scale an existing nascent technology would be free to compete at doing so.
It should be noted that even die hard capitalists are against IP restrictions [1] as they are a massive government investment in the market. So proponents of IP restrictions must reconcile their arguments in favor of this government intervention with their potential interest in free markets.
I mean it sure sounds like the old “science can’t explain how bumblebees fly, therefore science is incomplete or outright wrong” argument. Which is of course just false.
SequoiaHope4 hours ago
I replied so feel free to read that.
serf1 day ago
you have it entirely backwards; patents dont protect just-a-dude, they protect the corporation.
how?
just-a-dude doesn't have a team of patent attorneys sitting in his back office waiting for work.
fkyoureadthedoc1 day ago
On the other hand those same corporations can generate, file, and litigate more patents than just a dude could ever hope to.
It's 2007. Just-a-dude has a great idea, he notices customers to his website often buy just one item, so he'll let them do that with one simple click. What's this, he's just received a cease and desist? Sorry bro, Amazon patented that 10 years ago.
BolexNOLA1 day ago
I mean we’re basically getting the same result. Tons of businesses, not to mention patent trolls, constantly harass individuals and small businesses trying to get their foot in the door or just run a small, sustainable business. Hell forget my business failing, it’s possible I’ll never even get to try my idea out!
OutOfHere1 day ago
There is not much labor here. Anyone should be able to engineer a model by using deep learning over pictures that map raw images to their hyperspectral variants in various settings, including in adversarial settings that are intended to confuse. All you would need is a sufficiently large and diverse dataset.
brookst14 hours ago
I hate the new rhetorical use of the word “rent”
reaperducer11 hours ago
"Rent-seeking" is square 3B on my HN bingo card. See also:
Regulatory capture
Late-stage capitalism
Walled garden
An/The unreasonable...
So...
Period. Full stop
I Mean…
Streisand effect
orthogonal
trivial
non-trivial
{$person}'s Law | Axiom | Razor | Paradox
Objectively
Ship of Theseus
Gatekeeping
brookst23 minutes ago
It’s like the opposite of traditional tech buzzword bingo where you raise VC for blockchain crypto AI agents.
Instead, anything you don’t like is a rent-seeking late stage capitalism narrative.
danbruc14 hours ago
I do not understand how this could possibly work. From a camera with RGB filters we get essentially three different integrals over the spectrum per pixel, how would we recover the spectrum from that? Even assuming you can account for the spectrum of the light sources and the color filters in the camera - which should be doable with a color chart with known spectra - I do not see how you could go from three data points to a full spectrum without making assumptions about the possible object spectra.
EDIT: Okay, after going to the actual paper I at least get transmission mode - you photograph the color chart through the sample and this will of course imprint the absorption spectrum onto the know spectrum of the color chart and you can then look at the difference to the color chart without the sample in between. But I do not get the logic behind their reflectance mode.
abeppu1 day ago
I notice the article doesn't say anything about accuracy. This is not my area, but I think the _other_ hacky way to try to do spectroscopy with a phone is with a diffraction
grating (and maybe a box with a slit in it). Diffraction gratings are cheap, probably not so different from a specially-printed reference card. If you have a choice, which is better?
ted_dunning14 hours ago
The actual article does say quite a bit about accuracy.
csmoak1 day ago
diffraction grating wouldnt give you a controlled lighting environment (illuminant). they seem to handle that issue here by using a known spectral reference chart which might let them handle any normal lighting environment.
abeppu1 day ago
I would think in the same environment you would take images immediately before and after adding the sample.
sirtaknt1 day ago
I don't understand how from 3 independent values per pixel (RGB) they claim to derive 200+ independent values per pixel. Unless they are assuming a smooth "image" (all pixels the same RGB), perturbed only by the color card? Not exactly a camera then
ladberg1 day ago
They're not claiming to get that many values per pixel, they're getting that many values overall for the medium through which light passes between the card and the phone. The idea light comes from a source (e.g. sun), bounces off the various colors of the card and thus produces hundreds of different spectra, those all pass through a medium, and land on the phone camera. So you're getting one measurement consisting of hundreds of RGB values that each represent intensity of different spectra, and you combine it all together to get a single spectrogram.
ted_dunning14 hours ago
Did you read the actual article. They go into the method they use at length.
docfort1 day ago
More specific info on the reference card is available in the paper's supplemental information. https://ieeexplore.ieee.org/ielx8/83/10795784/11125864/supp1.... Basically, they used special paper with a pro-Canon inkjet, along with a special ICC color profile.
linsomniac1 day ago
Someone needs to build a phone that is leaning towards a tricorder; I'd buy that for myself and my kids. My Pixel 10 has a temp sensor on it, which is cool, but I've had minimal use so far.
I've always wanted to build a tricorder with my son, was just thinking about it last week when he was putting together a digital compass (with RasPi Nano, magnetic sensor, GPS, and LED light ring + OLED).
There are also cheap ($200-$400 range) usb-c thermal cameras specifically for phone use (they're cheap because they're just a sensor, the app on the phone is the "screen" and controls.) Great for narrowing down overheating hardware, and you can keep one in a pocket.
linsomniac1 day ago
I will say I have a FLIR C5 dedicated thermal camera, which I got used off Ebay for ~$300, and it's pretty useful to have around as a DIYer.
OutOfHere1 day ago
Thermal is interesting, but hyperspectral is much wider than just thermal. The two are not the same.
lawlessone5 hours ago
I want both :)
dylan6041 day ago
how does Cat's self repair policy compare to John Deere's? Then again, it's not far off from Apple's
DoctorOetker1 day ago
Every picture contains noise.
Every now and then we discover new traces left by the original light field (by using a fuller picture to model the image formation process, instead of an easier oversimplified one).
It makes one wonder how much of the noise is actually misinterpreted signal.
esafak1 day ago
This could improve chromatic adaptation of captured images. In other words, better results when changing the white point.
chaboud1 day ago
Funny enough, that’s what photographers are doing when they shoot a color checker chart (e.g., Munsell, Macbeth, X-Rite).
White balance is hard, in part, because the sensitivity bands of our vision and the camera sensors do not align. Take a look at fluorescent (or, better still, sodium vapor) light spectra for clarity on why this is a massive pain.
slwvx1 day ago
I was hoping that someone came out with a camera that not only had not only sensors for visible light, but for infrared and UV. It's just another color to add to the sensors; I think we have enough megapixels, seems like going for other bands is reasonable.
Scene_Cast21 day ago
I have a OnePlus 8 Pro with an IR camera. It's pretty nifty - nature photography looks cool, seeing through stovetops is neat (and seeing when they heat up), and VR things are also often playing around with IR (plastic transparent to IR, IR LEDs, etc).
I ended up having to flash Lineage, as there was some outrage that in a highly limited set of circumstances, thin see-through T-shirts became slightly more see-through and OnePlus disabled that camera in their later firmware updates.
chankstein381 day ago
You have to love when amazing innovations disappear just in case the lowest-quality rung of our society might misuse something... I'm pretty sick of being ruled based on the lowest common denominator.
zubiaur1 day ago
Back when I was in oil and gas, we were thinking of using modified mirror less cameras without and IR filter for vegetation density calculations. There were a few vendors that sold the UAVs and modified cameras.
Nowadays, there is a more mature ecosystem, with specialized drone mapping cameras tailored for the purpose.
For our use case, the micasense rededge would have been perfect.
seemaze1 day ago
I know many full size cameras have filters to specifically remove IR and UV from the images. Is this true for smartphones as well?
ACCount371 day ago
Yes.
CMOS image sensors are naturally sensitive to near IR. Early feature phones had no IR filters on their cameras - you could see an IR remote light up through them. But as people became more and more obsessed with smartphone camera quality, smartphones started to ship with those filters too. You get more "lifelike" colors that way.
Although in some multi-camera smartphones, one of the secondary cameras may lack an IR filter.
Tade01 day ago
One of mine definitely lacks such a filter because I was able to catch not only the remote, but also an electric stovetop while it was still heating up and its glow was barely visible with the naked eye.
chankstein381 day ago
Same! I want to be able to capture more of the spectrum already!
sirtaknt1 day ago
Some phones had near-IR camera (Pixel 4, Samsung S10) accessible via API. No "killer app" was found since then, 5+ years
ACCount371 day ago
iPhones also have a near-IR front camera, but that one is fully slaved to the FaceID system. Don't think anything in userland can access raw data from it.
grgergo1 day ago
There are lots of 3D scanning apps using "Face ID", like Heges: https://hege.sh/
ACCount371 day ago
Those rely on the depth maps, which can be accessed from userspace. But the depth maps are derived from IR camera footage, which is not accessible.
Ironically, older iPhones have better depth resolving capability overall. Apple sacrificed depth sensing performance in favor of smaller unit size in the newer ones.
ChrisRR15 hours ago
Hyperspectral imaging see things the human eye can't: Sure
Custom processors to accelerate generation of AI text: Go ahead
Slightly thicker to fit a bigger battery in: How dare you
mikewarot10 hours ago
This is a combination of things that almost breaks my brain, but it works, and it's brilliant!
First, you need to understand coded masks[1,2]. This is a way to use an LCD or other array to mask off parts of a scene so that very specific parts of it are sampled, but the rest aren't, to get a single, high resolution analog value. Then you switch to other masks, and get more values. You can then work backwards in the math through the known mask shapes, to get the original image with far fewer samples that would be required one at a time.
Think of the above as a 2d visual version of the Fourier transform[3,4]. This transform is used heavily to compress images throwing away most of the bits in an image without losing it's essence.
The analysis they're talking about uses a very specially printed card. It isn't just something generated with a standard 4 ink printer, each "dot" is a separate unique ink with tightly controlled spectral curves, these form a virtual version of the above masks. When you view these through a sample, it can then give an idea of the spectral response of the camera, and the liquid, by using the many different known response curves of the "dots" to work backwards, and generate the 1 dimensional very tight response curve of a hyperspectral imager, by figuring out where each "dot" is in the scene, then averaging that dot's intensity across the RGB values of the picture taken by the camera. Today's cameras have sufficient resolution and bit depth that you get the original bit depth (usually 8 bits) and an additional bit for each doubling of the number of pixels in a given "dot". This is degraded by Bayer pattern filters[5,6], and the nature of cameras, but it's not unrecoverable.
Like with Coded Masks, and Fourier transforms, you then take your high resolution analog values, and work backwards to get the things you want to measure.
Patent-pending... again someone trying to rent-seek a high-school physics fair idea. Measuring light absorption with a camera is almost as old as the camera.
Using a known reflectance chart in-scene to recover spectral information is a standard calibration technique.
What "investment" is patent law protecting here?
aDyslecticCrow1 day ago
What you're referring to is color calibration. This is spectroscopy. This is likley more of a chemistry paper than a engineering paper because the ink in the reference chart is doing some heavy lifting.
This is really cool and very clever. But i want to raise one thing.
> designed a special color reference chart that can be printed on a card
My rudimentary understanding of physics makes me suspect this sentence is a simplification.
A normal printer use Cyan Magenta Yellow Black to print. A photo of such a print would already destroy alot of spectral information for the same reason the individual rgb sensors do.
So i suspect those colored dots are a very careful and deliberate concoction of very particular inks with very specific spectral color bands.
I suspect alot of effort went into finding, mixing and algoritmically combining the right inks.
I'm guessing it works similarly to a how a narrow band florescent lamp makes only materials that reflect a very specific frequency be visible, which makea alot of prints and pigments look wierd. (If you do the opposite; use ink with very specific spectral band, you can instead measure the lamp)
Insanely clever. (Whatever they did)
Not just that, but it would presumably be sensitive to light emission spectra too. As inks can only reflect wavelengths of light that hit them, if the emission spectra has spikes or gaps - think LED or florescent - the reflected spectra will be a function of the light source[1].
Perhaps there's some accounting for this, and I'm curious to learn what it is, because it's a phenomenally complex problem.
1. You might think the sun is a standard source, but it's usually modulated by the atmosphere[2].
2. Unless you are in space.
> Perhaps there's some accounting for this, and I'm curious to learn what it is
The slip itself is a calibration reference, so a clean photo of it could serve to compensate for the lamp and camera and calculate how accurate the readings is for different parts of the spectrum. (But good wide spectrum light would be ideal for high precision readout)
You're also still limited to visible light because of the camera uv and ir filter, for which the sun is a decent reference.
oh, yes of course! Thank you :)
I don't think the sun is even a perfect source when you're in space, doesn't it have gaps in its emission spectra from the gasses that make it up?
You are quite right, I had no idea.
"Between around 10,000 nm (far infrared) and around 100 nm (deep ultraviolet), the spectrum of the Sun's spectral irradiance agrees reasonably well (though not perfectly) with that of a blackbody radiator at about 5,700K. That is about the temperature of the Sun's photosphere. The deviation from a perfect blackbody spectrum is due to many factors, including the absorption of light by constituents of the solar atmosphere, and the fact that the photosphere is not uniform, but has some hotter and some cooler regions, so that what is seen from the Earth is a composite spectrum of blackbody radiators at a range of different temperatures. About 99% of the total electromagnetic radiation coming from the Sun is in the ultraviolet-visible-infrared region."
https://acd-ext.gsfc.nasa.gov/anonftp/acd/daac_ozone/Lecture...
I was initially surprised too, but I think there is a some trick...
I'm not sure what happens on paper, but when you have ink disolved in water the abortion is not linearly proportional to the concentration, it's exponential. For example, consider a red ink and 5 magical selected frequencies and the absortions at 1% of concentration are: 99%, 99%, 50%, 10%, 1%
If you double the ink at 2% you get 99.99%, 99.99%, 75%, 19%, 1.99%
So increasing or decreasing the ink concentration may give you information of different frequencies, even with only one ink and only one sensor. In this case mostly about the 3rd and 4th. With more concentration you may kill all the light in the 3rd and measure the absortions ratio between the 4th and the 5th.
One problem I see here is how to order them, but I guess it's possible with a few sensors and a few inks. Each sensor sees all the frequencies, but with different weight. I'm not sure if it's possible to solve this, but perhaps you need some initial approximated model of the inks(???).
Now, when you put the inks on paper and have a unreliable light source and perhaps other technical problems, ...
In conclusion, I think it's possible to use different saturation and mixes of the inks to get different spectral distribution of the light that bounce on the card. Then use the three sensors to get three averages and try to use big linear algebra book to reconstruct what happens in between. But I should read the paper to be sure.
Based on supplementary material "We utilized a professional photographic inkjet printer (ImagePROGRAF PRO-1000, Canon), equipped with 11 ink cartridges and a chroma optimizer". Not your typical office printer but nowhere near as exotic as you might expect.
How special are the inks in the cartridge, though?
They are special in the sense that they are calibrated and QA'd for professional use
I have to assume it used inks available from Canon. Making your own inks is a research project on its own, so my guess is that would have been mentioned.
Wouldn't it be nice if they just told us so we didn't have to speculate? This is cool stuff and I'm glad I know about it but, as someone interested in this field of study, I'd love to try this out. But I guess I should stop being surprised when even a company like IEEE can't be bothered to write an article with any actual information. Just a bunch of simplified summarized crap.
At the bottom of the article is a link to the paper, which is open access.
I missed that! Thank you!
I would assume that with time, you could just print it on a generic $30 inkjet, and calibrate the card, sensor, and whitelte balance to a stored reference image.
It won't be as accurate, but it might be enough to offer some insights into whether liquid photographed in the article is in fact whisky, not urine (which to me seems to be a much more noble demonstration subject).
Ink is perfectly capable of being a phosphor, in which case it'll up or down convert wavelength X to wavelength Y.
My gut feeling is that finding enough very specific wavelength shifting inks would be harder. Perhaps its a mix though to get good readings in the faar edges between the rgb wavelengths.
I hope there is a research paper on this i can read.
I expect rolling shutters to be a far larger issue. Haven't read the article a second time, but non-uniform and inconsistent lighting is a huge challenge for this kind of work.
https://ieeexplore.ieee.org/ielx8/83/10795784/11125864/supp1...
We utilized a professional photographic inkjet printer (ImagePROGRAF PRO-1000, Canon), equipped with 11 ink cartridges and a chroma optimizer (PFI-1000 LUCIA PRO Ink, Canon), and used the manufacturer-recommended genuine paper (Photo Paper Premium Fine Art Smooth, Canon) for printing. To reproduce the desired reference colors for the spectral color chart, we also implemented a customized printing calibration process while maintaining the International Color Consortium (ICC) profile. The actual printed colors (output) showed notable distortions compared with the intended colors (input), which were particularly influenced by the type of paper (print sheet). For customized printing calibration, we mapped the exact relationship of the CIE xy chromaticity values between the digital color input and printed output values. After the printing process was completed, we measured the reflectance spectra of all reference colors from the printed spectral color chart (Fig. S1) using a spectrometer and a diffuse reflectance standard (equivalent to using CIE illuminant E). We confirmed that the CIE xy chromaticity values obtained from these measurements were in excellent agreement with the desired input values within the SWOP v2 gamut (Fig. 1(e)).
Printing can use so-called spot colors.
I similarly thought that just because they said print does not mean it was printed on someone's ink jet. I'd hate to see how many different Pantone colors might be necessary.
If you only need one card per 10,000 photos, then the cost of the card starts to look cheap compared to a spectrometer and its bulk.
This doesn't really seem like "hyperspectral imaging". I think the idea is having a reference colour chart of known emission characteristics and photographing it through a transparent substance gives you an idea of how much that substance attenuates each wavelength.
It's a cool trick if it works, but it seems very finicky and I guess would be limited to transparent/homogeneous liquids?
On top of that it only works in the VIS range, thanks to the filters in front of the camera sensors - and most of the interesting information is in UV or IR. VIS only contains information about a few elements. (see Fraunhofer lines https://en.wikipedia.org/wiki/Fraunhofer_lines)
In theory maybe you could build a version made of inks printed on a reflective mirror? And then you would hold the mirror so it reflected the object into the camera?
But that seems far more difficult. Precisely combining and applying combinations of inks to a mirrored surface sounds like a helluva manufacturing challenge.
I think ink and mirrors are kind of fundamentally incompatible.
Probably closer to what you're thinking about would be putting a bunch of tiny bandpass filters infront of a mirror, but in that case you can ditch the mirror entirely and just point the camera through the filter array.
A filter array right on-top of the sensor is how (the vast majority) of) CMOS cameras distinguish colour anyway.
> The new patent-pending technique
> “Every photo carries hidden spectral information waiting to be uncovered. By extracting it, we can turn everyday photography into science.”
And with our patent, extract rent from anyone who wants to do it!
That's a bit of a bad faith take. You were welcome to go spend the years(?) this chaps dedicated to putting together the research required to build this. If it works, let him enjoy the fruits of labour.
Sure, if he'd come up with this primarly using his own resources and time, but he discovered this whilst being paid to conduct research at a public university, a form of institution which is explicitly intended to disseminate knowledge. Society should enjoy the fruits of its investment.
Would the university have been willing to invest in the exotic printer and the labor to do the work without the potential upside of a patent?
It’s very easy to declare that someone else “should” do a bunch of work and spend a bunch of money for the altruistic benefit of society.
Patents do more than let you enjoy the fruits of your labor - the market already allows for that. Patents use the force of law to bar anyone else who might have discovered the same thing from building upon it.
Imagine you are just a dude, you did all this work, and go to "market".
You are just a dude, therefore business grows slowly.
You gather enough attention that some corporation with a lot of bling just goes and copies your thing.
Your business fails.
Once can just as trivially construct an argument demonstrating the issue with patents but the problem with this style of argument is that patents are not a simple thing. They have global far reaching effects. The government distributing a monopoly on information is a serious interference with the market, and due to patent harmonization efforts across the world, one person filing a patent in New Jersey affects even people in Kenya and Turkey and Thailand. The arguments for patents are often, as I see it, based on a deeply flawed understanding of the motivations of innovators and the affects of open information on innovation. For example most arguments in favor of patents cannot explain how open source works, and so are clearly incomplete or outright wrong.
>For example most arguments in favor of patents cannot explain how open source works, and so are clearly incomplete or outright wrong.
Can you clarify this? Just curious about what you mean.
I often speak to people who say that no one would innovate without government-secured monopoly on information, and this is clearly false as open source works without this monopoly. As such it seems to me that people who make that claim don’t understand how innovation works.
Often I say that if we phased out patents and other IP restrictions, then investments would not stop but they would change from less frequent large investments to more frequent small investments. As long as designs can stay secret until release, there will always be first mover advantage and brand recognition. But you might get smaller investments to build out manufacturing for the next year, rather than bigger longer term investments. The flip side is that stagnant innovators who got lucky once will be subsumed by more agile competitors who can better deliver those innovations to market.
Thus investment and innovation wouldn’t stop - markets still ensure certain advantages for innovators - but the nature of investment and innovation wouldn’t shift towards more incremental moves and more diverse actors. A major upside to this is that those best suited to scale an existing nascent technology would be free to compete at doing so.
It should be noted that even die hard capitalists are against IP restrictions [1] as they are a massive government investment in the market. So proponents of IP restrictions must reconcile their arguments in favor of this government intervention with their potential interest in free markets.
[1] https://youtu.be/GZgLJkj6m0A
I mean it sure sounds like the old “science can’t explain how bumblebees fly, therefore science is incomplete or outright wrong” argument. Which is of course just false.
I replied so feel free to read that.
you have it entirely backwards; patents dont protect just-a-dude, they protect the corporation.
how?
just-a-dude doesn't have a team of patent attorneys sitting in his back office waiting for work.
On the other hand those same corporations can generate, file, and litigate more patents than just a dude could ever hope to.
It's 2007. Just-a-dude has a great idea, he notices customers to his website often buy just one item, so he'll let them do that with one simple click. What's this, he's just received a cease and desist? Sorry bro, Amazon patented that 10 years ago.
I mean we’re basically getting the same result. Tons of businesses, not to mention patent trolls, constantly harass individuals and small businesses trying to get their foot in the door or just run a small, sustainable business. Hell forget my business failing, it’s possible I’ll never even get to try my idea out!
There is not much labor here. Anyone should be able to engineer a model by using deep learning over pictures that map raw images to their hyperspectral variants in various settings, including in adversarial settings that are intended to confuse. All you would need is a sufficiently large and diverse dataset.
I hate the new rhetorical use of the word “rent”
"Rent-seeking" is square 3B on my HN bingo card. See also:
It’s like the opposite of traditional tech buzzword bingo where you raise VC for blockchain crypto AI agents.
Instead, anything you don’t like is a rent-seeking late stage capitalism narrative.
I do not understand how this could possibly work. From a camera with RGB filters we get essentially three different integrals over the spectrum per pixel, how would we recover the spectrum from that? Even assuming you can account for the spectrum of the light sources and the color filters in the camera - which should be doable with a color chart with known spectra - I do not see how you could go from three data points to a full spectrum without making assumptions about the possible object spectra.
EDIT: Okay, after going to the actual paper I at least get transmission mode - you photograph the color chart through the sample and this will of course imprint the absorption spectrum onto the know spectrum of the color chart and you can then look at the difference to the color chart without the sample in between. But I do not get the logic behind their reflectance mode.
I notice the article doesn't say anything about accuracy. This is not my area, but I think the _other_ hacky way to try to do spectroscopy with a phone is with a diffraction grating (and maybe a box with a slit in it). Diffraction gratings are cheap, probably not so different from a specially-printed reference card. If you have a choice, which is better?
The actual article does say quite a bit about accuracy.
diffraction grating wouldnt give you a controlled lighting environment (illuminant). they seem to handle that issue here by using a known spectral reference chart which might let them handle any normal lighting environment.
I would think in the same environment you would take images immediately before and after adding the sample.
I don't understand how from 3 independent values per pixel (RGB) they claim to derive 200+ independent values per pixel. Unless they are assuming a smooth "image" (all pixels the same RGB), perturbed only by the color card? Not exactly a camera then
They're not claiming to get that many values per pixel, they're getting that many values overall for the medium through which light passes between the card and the phone. The idea light comes from a source (e.g. sun), bounces off the various colors of the card and thus produces hundreds of different spectra, those all pass through a medium, and land on the phone camera. So you're getting one measurement consisting of hundreds of RGB values that each represent intensity of different spectra, and you combine it all together to get a single spectrogram.
Did you read the actual article. They go into the method they use at length.
More specific info on the reference card is available in the paper's supplemental information. https://ieeexplore.ieee.org/ielx8/83/10795784/11125864/supp1.... Basically, they used special paper with a pro-Canon inkjet, along with a special ICC color profile.
Someone needs to build a phone that is leaning towards a tricorder; I'd buy that for myself and my kids. My Pixel 10 has a temp sensor on it, which is cool, but I've had minimal use so far.
I've always wanted to build a tricorder with my son, was just thinking about it last week when he was putting together a digital compass (with RasPi Nano, magnetic sensor, GPS, and LED light ring + OLED).
Take a look at the phyphox app - https://phyphox.org/
Also Arduino Science Journal: https://www.arduino.cc/education/science-journal
Wow! That app is amazing thanks for sharing
This is brilliant.
Caterpillar has a smartphone with a thermal camera. The price isn't far off the the price of the most expensive smartphones
https://cat.smartwalkie.com/store/products/cats62pro
There are also cheap ($200-$400 range) usb-c thermal cameras specifically for phone use (they're cheap because they're just a sensor, the app on the phone is the "screen" and controls.) Great for narrowing down overheating hardware, and you can keep one in a pocket.
I will say I have a FLIR C5 dedicated thermal camera, which I got used off Ebay for ~$300, and it's pretty useful to have around as a DIYer.
Thermal is interesting, but hyperspectral is much wider than just thermal. The two are not the same.
I want both :)
how does Cat's self repair policy compare to John Deere's? Then again, it's not far off from Apple's
Every picture contains noise.
Every now and then we discover new traces left by the original light field (by using a fuller picture to model the image formation process, instead of an easier oversimplified one).
It makes one wonder how much of the noise is actually misinterpreted signal.
This could improve chromatic adaptation of captured images. In other words, better results when changing the white point.
Funny enough, that’s what photographers are doing when they shoot a color checker chart (e.g., Munsell, Macbeth, X-Rite).
White balance is hard, in part, because the sensitivity bands of our vision and the camera sensors do not align. Take a look at fluorescent (or, better still, sodium vapor) light spectra for clarity on why this is a massive pain.
I was hoping that someone came out with a camera that not only had not only sensors for visible light, but for infrared and UV. It's just another color to add to the sensors; I think we have enough megapixels, seems like going for other bands is reasonable.
I have a OnePlus 8 Pro with an IR camera. It's pretty nifty - nature photography looks cool, seeing through stovetops is neat (and seeing when they heat up), and VR things are also often playing around with IR (plastic transparent to IR, IR LEDs, etc).
I ended up having to flash Lineage, as there was some outrage that in a highly limited set of circumstances, thin see-through T-shirts became slightly more see-through and OnePlus disabled that camera in their later firmware updates.
You have to love when amazing innovations disappear just in case the lowest-quality rung of our society might misuse something... I'm pretty sick of being ruled based on the lowest common denominator.
Back when I was in oil and gas, we were thinking of using modified mirror less cameras without and IR filter for vegetation density calculations. There were a few vendors that sold the UAVs and modified cameras.
Nowadays, there is a more mature ecosystem, with specialized drone mapping cameras tailored for the purpose.
For our use case, the micasense rededge would have been perfect.
I know many full size cameras have filters to specifically remove IR and UV from the images. Is this true for smartphones as well?
Yes.
CMOS image sensors are naturally sensitive to near IR. Early feature phones had no IR filters on their cameras - you could see an IR remote light up through them. But as people became more and more obsessed with smartphone camera quality, smartphones started to ship with those filters too. You get more "lifelike" colors that way.
Although in some multi-camera smartphones, one of the secondary cameras may lack an IR filter.
One of mine definitely lacks such a filter because I was able to catch not only the remote, but also an electric stovetop while it was still heating up and its glow was barely visible with the naked eye.
Same! I want to be able to capture more of the spectrum already!
Some phones had near-IR camera (Pixel 4, Samsung S10) accessible via API. No "killer app" was found since then, 5+ years
iPhones also have a near-IR front camera, but that one is fully slaved to the FaceID system. Don't think anything in userland can access raw data from it.
There are lots of 3D scanning apps using "Face ID", like Heges: https://hege.sh/
Those rely on the depth maps, which can be accessed from userspace. But the depth maps are derived from IR camera footage, which is not accessible.
Ironically, older iPhones have better depth resolving capability overall. Apple sacrificed depth sensing performance in favor of smaller unit size in the newer ones.
Hyperspectral imaging see things the human eye can't: Sure
Custom processors to accelerate generation of AI text: Go ahead
Slightly thicker to fit a bigger battery in: How dare you
This is a combination of things that almost breaks my brain, but it works, and it's brilliant!
First, you need to understand coded masks[1,2]. This is a way to use an LCD or other array to mask off parts of a scene so that very specific parts of it are sampled, but the rest aren't, to get a single, high resolution analog value. Then you switch to other masks, and get more values. You can then work backwards in the math through the known mask shapes, to get the original image with far fewer samples that would be required one at a time.
Think of the above as a 2d visual version of the Fourier transform[3,4]. This transform is used heavily to compress images throwing away most of the bits in an image without losing it's essence.
The analysis they're talking about uses a very specially printed card. It isn't just something generated with a standard 4 ink printer, each "dot" is a separate unique ink with tightly controlled spectral curves, these form a virtual version of the above masks. When you view these through a sample, it can then give an idea of the spectral response of the camera, and the liquid, by using the many different known response curves of the "dots" to work backwards, and generate the 1 dimensional very tight response curve of a hyperspectral imager, by figuring out where each "dot" is in the scene, then averaging that dot's intensity across the RGB values of the picture taken by the camera. Today's cameras have sufficient resolution and bit depth that you get the original bit depth (usually 8 bits) and an additional bit for each doubling of the number of pixels in a given "dot". This is degraded by Bayer pattern filters[5,6], and the nature of cameras, but it's not unrecoverable.
Like with Coded Masks, and Fourier transforms, you then take your high resolution analog values, and work backwards to get the things you want to measure.
[1] https://www.youtube.com/watch?v=_ezhdhHNku0
[2] https://en.wikipedia.org/wiki/Coded_aperture
[3] https://en.wikipedia.org/wiki/Fourier_transform
[4] https://www.youtube.com/watch?v=spUNpyF58BY&pp=ygURZm91cmllc...
[5] https://www.youtube.com/watch?v=LWxu4rkZBLw&pp=ygUMYmF5ZXIgZ...
[6] https://en.wikipedia.org/wiki/Bayer_filter
Patent-pending... again someone trying to rent-seek a high-school physics fair idea. Measuring light absorption with a camera is almost as old as the camera.
Using a known reflectance chart in-scene to recover spectral information is a standard calibration technique.
What "investment" is patent law protecting here?
What you're referring to is color calibration. This is spectroscopy. This is likley more of a chemistry paper than a engineering paper because the ink in the reference chart is doing some heavy lifting.