Using different shades of fruit as pixels
The USDA Pomological Watercolor Collection is a huge collection of watercolor images of fruits and nuts that was created between the 1880s and 1930s. The collection spans about 7.000 images with 2-4 individual fruit elements per image. My idea was to use all those approximately round elements as pixels to reproduce normal photographs using those fruit pixels. This is the write-up of how I did it.
Acquiring the USDA Pomological Watercolor Collection
The first step was to obtain all images in the collection. The collection is described at a designated page by the National Agricultural Library of the U.S. Department of Agriculture. The page links to the actual collection page, where each image can be viewed individually. Sadly, there is no way to download all images at once, and the page is also programmed in a way that makes crawling difficult.
Luckily, someone already mirrored the collection on Wikimedia, and so I opted to crawl all images from there.
I did the crawling 2 years ago for another project, so I am not sure if there might be an option to download the collection as a whole today.
One last thing to note is that the whole collection is in the public domain, so we can work with it freely.
Converting the Images to Individual Assets
The images of the collection often contain several fruits in one image. If we want to use the fruits as individual pixels, it would be ideal to separate these into their own files. There are also handwritten notes on a lot of images that we are not interested in.
As a first step, I batch processed all images with the tool rembgto remove the backgrounds. This worked great and gave me a folder full of PNGs of fruits with transparent backgrounds.
However, the fruits were still together in one image, so the next step was to separate them. Because they were now clearly separated on a transparent background, that was quite easy. I used Rust, the crate imageproc, and the function connected_components to identify the individual fruits and saved them to separate files. I also scaled them down and converted them to the Quite OK Image Format (QOI) so I could read them faster from file when using them later. In a previous iteration, I used PNG for saving the assets and didn't downscale the images, and that was painfully slow to use.
I wrote a little tool to automatically work through all 7.000 images and ended up with around 11.000 individual assets. rembg had sometimes failed to correctly separate out the background, and this gave me fruits that were mostly grey as assets. I reworked the tool and added some simple filters to discard those.
Colors
In order to use these images as pixels, we have to determine their dominant color. A few weeks ago I read this article on identifying the main color of an image and used some of the ideas presented. The core idea is that averaging all colors is a bad idea that will often leave you with muddy colors as a result. The author also argued that working in the sRGB color space further worsens the result. He proposed using the Oklab color space.
The core idea of the author is to not use an averaging approach but K-Means Clustering to identify N dominant colors of the image and then pick one. I tried that, but K-Means Clustering was really unperformant (at least when I used it with Python in a Jupyter Notebook). I opted to use a weighted average, where the intensity of the color was the weight, so more intense colors counted more towards the average. That worked for my use case. The image below shows how all the assets are placed in the Oklab color space using this method.
Reflecting now, while doing this write-up, I had the idea that by using K-means clustering, I would maybe be able to place these things differently and maybe use more of the color space. If that translates to better end results (whatever better means in this case), remains to be seen. Even though K-Means Clustering was unperformant with Python, I am now using Rust and downscaled images, so maybe it can work.
Intermission - Ideas that seemed Smart but weren't
Before I tell you about the final step to create images with the fruit pixels and show you results, I want to do a little intermission where I tell you about the ideas I thought were smart but didn't work out.
When first creating images with the fruit pixels, I placed them in a grid. I was unsure if I liked the grid look or maybe wanted a more chaotic look. I tried placing points randomly in a 2D canvas and placing my fruit pixels at those locations; however, I didn't like the look. In the end I preferred the natural shape of the fruits contrasted with the forced order of the grid.
I also tried to make Floyd-Steinberg dithering work. I had never used it before but knew it could be used to obtain nicer results when reducing the colors in an image, as opposed to simply using the most similar color of the reduced color set to the original color set at each pixel. However, I made each fruit 20-40 px large to get a nice balance between seeing the individual fruits and also being able to still see the whole image. At this scale, Floyd-Steinberg dithering had no effect, and so I didn't use it in the end.
Creating Images from the Assets
Creating Images from the Assets
With all the things that didn't work out of the way, let's focus on what worked and how I got from a folder full of fruit assets and an input image to my final results.
The first step was to load all images into my program. Each image was analyzed for the dominant color using the method explained above and placed in a list. After that, a grid was placed on the original image, and for each pixel in the original image at a grid point, the best fruit asset by dominant color was identified. The fruit was placed on a new empty canvas, and thus the new image was built up. The results looked promising but also had two major flaws. Even though there were many assets with bright reds and greens, the algorithm favored the duller colors. Also, the algorithm often seemed to pick the same assets, making for a very repetitive image.
To combat the problem with the duller colors, I added a vibrancy bias. To check if an asset can represent an original color, the system checks the Euclidean distance in the Oklab color space between the original color and the dominant color of the asset. The vibrancy bias gets multiplied with the vibrancy of an asset and then subtracted from the overall distance. Thus, the system can be tuned to prefer more vibrant colors.
To combat the problem with the repetition, I added an overuse penalty. The system keeps track of how often a tile was already used and calculates an excess score quantifying if the tile is overused. The score then gets multiplied by the overuse penalty and added to the distance calculation between a candidate asset and the original color, meaning overused assets become more unattractive to pick. I also randomly rotate the picked asset when placing, so even the same assets give a different impression.
The last thing I fixed was adding some pictures of blue candies to the assets, because i felt this would make the images more interesting.
And so, without further ado:
Further work
The algorithm is still not perfect. There are still some grey transparent assets in my 11.000 assets that lead to black spots in the final images; for example, the floppy drive of the Macintosh. This needs to be fixed.
Loading and analyzing the images into one list takes the most time when running the tool. This can be done once and then cached for each new tool run.
I also plan to revisit the K-means approach for dominant color calculation in Rust to see if it will give me better results.