3D Gaussian Point Art

Generative AI has demonstrated significant potential in creating images that mimic traditional art. With this project, I wanted to explore how AI could transform traditional artworks into an entirely new experience.

The technique I employed is a modified version of 3D Gaussian Splatting, a cutting-edge AI-based method for reconstructing 3D scenes from multiple images, replacing anisotropic splats with isotropic points. Instead of using real-world photographs, I curated a set of related artworks, each serving as a constraint in a given viewing direction on a volume of randomly distributed 3D Gaussian points. Through iterative optimization, the AI refines the position, size, color, and opacity of these points, aligning them with the input artworks from their respective perspectives.

Unlike traditional pointillism, which relies on the optical blending of static dots on a 2D surface, this method constructs a dynamic 3D composition, where points shift and adjust based on perspective, introducing depth and movement that conventional art lacks.

My role in the creative process involves tuning hyperparameters and experimenting with different loss functions to guide the AI toward generating the desired outcome. Additionally, I instructed the AI to minimize the number of points, encouraging it to discover patterns and connections among the artworks. This also raises intriguing questions about the role of AI in creative expression, as the system autonomously refines itself toward artistic cohesion while being guided by human input.

I deliberately chose artworks with similarities in color, style, and subject matter—not because the AI requires them, but because they add meaning for human perception. It’s fascinating to observe how the AI can transform even non-impressionist pieces into impressionistic renditions.

I also experimented with generative AI to create the input artworks. Impressionistic pieces generated by Midjourney were incorporated into the 3D Gaussian optimization process, allowing one AI model to feed its output into another, effectively completing the creative cycle through generative methods.

To enable interactive exploration in a web browser, I adapted a Three.js-based Gaussian splat renderer. This renderer automatically cycles through predefined viewpoints while allowing users to navigate freely and explore the 3D artwork from multiple perspectives.