Have a damaged painting? Restore it in just hours with an AI-generated “mask”
news.mit.edu71 points by WithinReason 3 days ago
71 points by WithinReason 3 days ago
I will reserve judgement on this for the simple fact is that att conservation has been responsible for a huge amount of Art vandalism.
The 'restoration' of the cistine chapel ceiling was funded by a Japanese tv company. The cheapest approach was chosen which assumed that michelangelo made absolutely no corrections to his fresco using applied paint. It is perfectly obvious that this was a mistaken assumption, in the process removing many of the artists original work. I can upload some slides later if anyone is interested.
In victorian times many classic sculptures were scrubbed of their original paint and their stonework bleached, just in order to serve the tastes of the time.
And let us not forget that modern conservators will add or remove elements according to the clients taste. Eg change the flag of a ship from British to American.
In that case they explicitly says that the process is non destructive and reversible. Essentially a wrap over the original artwork
> The restoration is printed on a very thin polymer film, in the form of a mask that can be aligned and adhered to an original painting. It can also be easily removed. Kachkine says that a digital file of the mask can be stored and referred to by future conservators, to see exactly what changes were made to restore the original painting.
I wonder whether it's still worthwhile to replaced a yellowed varnish varnish on old painting, just to be sure that it doesn't degrade further (with the assumption that historical varnish are somewhat lesser than modern one, which I really don't know if it's the case).
I would assume the mask only covers the damaged areas. This means it replaces manual retouching, so you still need to clean the original painting and remove the varnish in order to get at the original colors, and as part of the usual cleanups (removal of old conservation, stabilisation, infill). The article specifically says that the film is adhered to and via a varnish layer.
Makes me wonder how it would handle heavy impasto tho.
Obviously it can't handle heavy impasto. With heavy impasto, people usually literally sculp the surface using some material to math the rest of the painting and then paint on top of this. Even with minimal impasto, they will often try to mimic the texture by imprinting various tools into the material, mimicking for example brushes used by the original author. This printing method can't do this at all, and the filter might struggle when the painting is too uneven.
Otherwise this might be an interesting technique, if the result can match the color and texture of paint perfectly. I can see it being used for some low priority paintings. There are much more paintinga that need restoration than people with necessary skills, so this could save of them, as it will be more viable to fix them. The infilling is usually just a small part of the entire process, but usually the most difficult wrt how skillful the conservator must be. You must be able to match colors and style perfectly, and there are huge differences in how fast this process is depending on the skills of the painter.
> I wonder whether it's still worthwhile to replaced a yellowed varnish varnish on old painting, just to be sure that it doesn't degrade further (with the assumption that historical varnish are somewhat lesser than modern one, which I really don't know if it's the case).
I would say that we should just live with the difference. There is a near guarantee that our experience of paintings as they exist today will always be significantly different to how they would have been experienced when they had been made, regardless of what we do. The factors are:
- Yellowing of oil/pigment/varnish. This is pretty much inevitable, even in modern paints.
- Fugitive colors. Most high-color pigments tended to fade significantly over time. Degas was once seen crying in front of a Delacroix, upset at how badly the colors had faded, even in his lifetime (It might have been Manet... can't locate the relevant anecdote online).
- Excessive varnishing. It used to be so that paintings received regular varnishing by their owners. Over time, the layers of varnish would build up to a ridiculous degree.
- Lighting. For me this is probably the most annoying factor. A general rule is that paintings should be lighted in the same light that they were painted. The values go to hell if a painting is lighted too brightly. There are many so-called hanging experts, who always light too strongly, even in respected national collections.
Semi-related anecdote: The first time I went to the Rodin museum I was struck by how badly one of the busts had been lighted. Whilst the guards were not looking, I moved it to a more agreeable position. I came back one year later to find it was still in that position.
As for using modern varnishes... I am not expert, but AFAIK one factor that accounts for the amazing longevity of old paintings was the compatibility between their many layers. The canvas, the primer, the paint and the medium were all derived from the same plant: flax. On top of that would sometimes be laid stand oil, again derived from flax. If this continuity is broken, all manner of problems might arise, most of which would be long term issues such as cracking and flaking.
> AFAIK one factor that accounts for the amazing longevity of old paintings was the compatibility between their many layers. The canvas, the primer, the paint and the medium were all derived from the same plant: flax.
AFAIK that's largely untrue in the general case:
- Canvas painting only appeared in the 15th century, and took a while to spread, paintings before the 17th century (16th in some locations) tend to be on wood panels e.g. raphael and da vinci painted almost exclusively on board.
- Canvas is almost always coated with gesso which at the time would use rabbitskin glue as binder, no flax there.
- While flaxseed was the most common drying oil (in europe), walnut, poppyseed, and safflower, were also in use. And additives were usually mixed in to manage the viscosity of the paint, so even in the "best" case it's not like the paint would be just pigments in linseed oil unless it's a very early oil painting, which wouldn't have been on canvas.
- As for varnishes, not only was flax not the only drying oil for oil varnishes, varnishes could also be "spirit" varnishes (with a resin dissolved into a solvent like alcohol or turpentine), waxes were also sometimes used as or in varnishes.
True… what I said only applies to canvas paintings. I should have made that clear.
The longevity of murals is easy to account for: the paint is applied to wet plaster, in that way becoming part of the wall. That is why the murals of Pompeii survived.
> Canvas is almost always coated with gesso which at the time would use rabbitskin glue as binder, no flax there.
You are right about the rabbit skin glue, but wrong about the gesso. As I recal, traditional Gesso is a mix of glue plus titanium white powder and is very brittle, generally unsuited to a flexible support such as canvas. A canvas painter would more likely use something flexible like a mix of pigment and rabbit skin glue, or pigment and egg protein or pigment and oil.
I thank god for modern primers. Using modern primers, I can prime a canvas in two days. An oil based oil primer could take months to dry.
And don’t forget the restoration attempt on the Ecce Homo portrait[0]
[0] https://en.wikipedia.org/wiki/Ecce_Homo_(Garc%C3%ADa_Mart%C3...
Honestly, that's a lot better than the original. We should restore more art that way.
It changes the medium. Even if it works perfectly, doesn't alter luminosity and perfectly preserves the art under the wrap, it's still shrink wrapped art and alters the experience of viewing the work.
Perhaps.. have you seen the 3d replica's in the van Gogh museum? The film could also have a 3d printed version.
The restoration method is more interesting than the AI, but claiming it uses AI is probably necessary to get funding for it.
Yes, it seems like the same method (a thin polymer mask of which a computer record exists) could be used in a completely human-driven process. Just allowing restorers to try a few different things to see what works would probably speed up their process immensely.
There's probably some other grant money they can get down the line for a completely AI-less process, but obviously there's a lot more funding for AI than for art restoration.
It's conceptually similar to current restoration techniques, but, the real innovation is in doing the whole thing as one piece ahead of time rather than requiring somebody to get in there with tiny brushes after a transparent base layer is completed.
It's also worth noting that this doesn't help with the first half of most restorations, which is removing crud and wear and often painstakingly undoing previous alterations or low-quality restorations. There's actually a pretty long history of paintings being altered to fit current fashions or otherwise mucked with in ways that make modern art historians cringe.
As far as I understand this only replaces the retouching phase. A human needs to do that in place in order to color and pattern match (and it's already hard enough that way). And I do not know but would assume the retouching paints can be removed without affecting the isolation layer if the conservator realises they went in the wrong direction earlier and can’t easily recover.
They could probably just tell people they're "using AI" and then just use whatever methods work best. It's not like investors are doing much more DD than listening for that phrase.
I would’ve argued against this so hard if it weren’t for that NIKOLA company advertising their trucks running on an “HTML supercomputer” and still getting millions in funding.
Unless the result is a form of protection to the painting then why not "just" reprint the whole painting and not touch the original at all?
Looking at a oil painting in real life is a vastly different experience than a looking at a print. Oil paintings are for a start 3D (textured). Colour reproduction is also difficult.
No thank you, that would destroy the value and art of the painting.
Funny aside, they have an odd ligature to for “fi” on that site. It shows up in iOS Safari at least.
> Kachkine acknowledges that, as with any restoration project, there are ethical issues to consider, in terms of whether a restored version is an appropriate representation of an artist’s original style and intent. Any application of his new method, he says, should be done in consultation with conservators with knowledge of a painting’s history and origins.
A sort of interesting thought, “what was the artist’s intent, should be recover the painting,” is a well known question nowadays. It would be sort of funny if current artists would just write down what sort of restoration plans they are ok with. I wonder how many would say “just do what you will do people can enjoy it.”
Although, one could argue maybe that the damage which occurs to artwork as it ages also tells a historical story. Perhaps that story doesn’t just belong to the artist, and so restoring the work could be questionable even with their permission. I’m sure this is well-trod ground.
Why not just generate a new painting with AI? How much of the painting has to be covered with a mask before it’s not a painting anymore?
That’s not exactly a gotcha. Art conservators have been grappling with variants of that question for hundreds of years.
The answer might just be: What's done is done. Conserving what's left is the best you can hope for.
> "Because there's a digital record of what mask was used, in 100 years, the next time someone is working with this, they'll have an extremely clear understanding of what was done to the painting"
Reminds me of https://xkcd.com/1683/
Whatever digital record format they use, they will probably need to rewrite it once every few years to ensure that it's still readable.
but most of the lift looks mechanical : masking, polymer filling, pigment blending. if ai's involved, it's probably color inference or mask prediction. nothing wrong with that, just feels overstated. curious if they benchmarked against expert restorers on visual fidelity. and who signs off on historical accuracy here? ai or the human behind the brush?
I hope the AI will not generate hands with seven fingers ...
That was a problem a year or two ago. New AI models have gotten past that point now.
openai native 4o image generation generates 4 fingers instead.
Is that a joke? Because 4o image generation (assuming you click "Image Generate" which uses gpt-image-1) EASILY handles rendering hands with the proper number of fingers even without specifying something like that.
If anything, it's actually MORE difficult to generate hands with an improper number of fingers. Apologies to Count Rugen.
The tweet about gpt 4o image generation capabilities release had 4 fingers.
I'd like to see that tweet if you have it. I also don't remember the intro page they put out having any such issue:
Look at the hands in "selfie view of the photographer, as she turns around to high five him".
Hmm, I agree it's not a good high-five - their hands aren't really centered against each other, but her raised right-hand is oriented at an angle where her thumb and index finger are kind of obscured since her pinky finger is closest to the camera.
I thought the man's hand was either missing a pinky finger or had an abnormally short one.
Here's another one, this tweeted image shows either six fingers or an abnormally large middle finger: https://x.com/OpenAI/status/1904602845221187829