Solo Developer Builds AI Cutout Tool That Stands Up To Adobe

Hendrik Kueck is the solo indie iOS developer of the app Juxtaposer, and his latest update includes an AI-based machine learning feature called Magic Crop that can look as powerful and effective as what Adobe offers in Photoshop on iOS and desktop.

Originally released in 2008, Juxtaposer is a photo compositing app that allows you to easily create photo montages from two or more images. What this typically means is cutting out elements from one or more images and adding them to a background image.

For the last year and a half, Kueck has been working on what ultimately became the Magic Crop feature.

Magic Crop has the user place a rectangle tightly around the image element that they want to have cut out. This rectangle, along with the image, is fed into the app’s Machine Learning model which then computes a segmentation mask (a black and white image that controls which pixels are made transparent).

The Machine Learning model runs on the user’s iOS device, which Kueck says he did to preserve privacy as no photo data is being sent to the cloud. The computation takes about 140 milliseconds on an iPhone 11, and 120 milliseconds on an iPhone 23. Kueck thanks CoreML and the Neural Engine on Apple’s recent devices for that speed. To the user, it feels instantaneous.

“The fact that a model this complex can run on a phone, and as fast as it does is just mind-boggling!”

Kueck tells PetaPixel that the model was developed and trained on a Linux PC using the PyTorch Machine Learning framework, and then converted to Apple’s CoreML afterward.

“It was trained using about 300,000 example pairs of a photo and corresponding black and white image mask indicating the pixels that belong to a particular object (or subject) in the photo,” Kueck says. “From these examples, the model learned how to cut out not just people and animals, but all kinds of objects.”

This kind of object isolation looks very similar to what Adobe released in Photoshop 2020 as part of its Object Selection tool. When this was pointed out to Kueck, he admitted he wasn’t aware of Adobe’s feature.

“I am embarrassed to admit that until just now I did not know about the ‘Object Selection’ tool that Adobe added in Photoshop CC 2020 and the ‘Smart Cutout’ tool in Photoshop Express on iOS, released a week ago,” he says. “These tools are actually very similar to the Magic Crop feature I added in Juxtaposer. They all are using a Machine Learning model and use it to cut out objects based on a rectangle drawn around the object (Photoshop on the desktop supports lasso selection also).”

It should be noted that Kueck claims he was developing his own solution to AI-powered photo cut-outs before Adobe released theirs, and was still working on his when the software giant pushed its update to the desktop. It’s likely fair to assume both he and Adobe were working on their own versions of this machine learning algorithm at the same time, unaware of each other.

After initial conversations with PetaPixel, Kuech was curious to see how his own algorithm stacked up to what Adobe created. He decided to compare them against one another, and what he found was a healthy mix of results.

“The first thing I found out is that the Object Selection tool in Photoshop on the desktop, and in Photoshop on the iPad are using the same Machine Learning model, and thus produce essentially the same results,” he told PetaPixel. “The ‘Smart Cutout’ (or ‘SmartCut’) tool in Photoshop Express on iOS however is using a different (most likely simpler) model, producing different results.”

Kueck says he compared Magic Crop to these machine-learning-based segmentation algorithms in Photoshop and PS Express on three different images, categorized by what he saw as easy (a portrait of Barack Obama), medium (a photo of a metal kettle), and highly-challenging (a photo of a woman holding a beer) for the algorithms to compute.

“I’ll let you draw your own conclusions,” he said after sharing the results. “But here are a few of my observations.”

“On the Obama portrait, the PS Express algorithm actually has the best results. Specifically, it handles the sharp corners at the ears the best. Photoshop’s Object Selection tool handles these the worst and for some reason cuts off a small bit of Obama’s suit jacket in the bottom left. Juxtaposer’s Magic Crop shaves some pixels off his hair and ears.”

With the medium difficulty image, Kueck says he thinks his app competes better.

“The kettle photo example is a close call between Magic Crop and the Photoshop Object Selection tool, but I would say Magic Crop is doing slightly better as its result is the only one with a clean boundary on the bottom,” Kueck argues. “Otherwise, it does miss little bits on the left hinge, whereas the Object Selection tool cuts off a bit of the spout. PS Express completely fails to cut out the empty space between handle and kettle and thus places a distant third.”

In this last, most challenging example, Kueck feels his app shows its mettle: While none of them are perfect, the results from Magic Crop are impressive.

“The most challenging example, Juxtaposer’s Magic Crop is the clear winner (though the result is certainly not perfect). It handles her feet very well, and does by far the best job with the extremely challenging left shoulder over the similarly colored background,” he says.

Kueck was honest about the limitations of his technology, and made sure to clarify what his features can do based on the competition and says that one other difference worth pointing out is that Magic Crop is currently a one-shot tool.

“Juxtaposer is obviously a much simpler and more focused app than Photoshop,” he says.

“Refinements to the mask can be made afterwards using Juxtaposer’s Erase and Unerase brush tools, but not using Magic Crop itself (other than adjusting the rectangle and trying again). Photoshop’s Object Selection tool can also be used to add or subtract from an existing mask.”

Kueck says that he currently has no plans to bring his Juxtaposer app to the desktop, but does see a future for the Magic Crop tool specifically there.

“I have vague potential plans to build another app around the Magic Crop functionality, and the idea would be to target both iOS and macOS with that,” he tells PetaPixel.

What is worth noting is how well Kueck’s algorithm competes with the ones developed by the largest creative software company in the world. While he has had help in the past, Kueck is currently a one-man show, and seeing the capability of his app standing in the same ring as Adobe is impressive. Further, when looking at side by side comparisons, the fact that in some cases his algorithm shows superior results to a full desktop application created by hundreds of developers is nothing short of incredible. While the use case for Kueck’s technology is perhaps more limited than what many do with Photoshop, what he has created by himself is nonetheless worthy of praise.

You can try out Juxtaposer on iOS for $3, and the Magic Crop feature shown here was just recently added at no extra cost.

from PetaPixel https://ift.tt/32Wih8V
via IFTTT