Three of the original stick puppets – Kaspar, der Polizist and die Moral – from the collection of the Museum für Gestaltung Zürich have been photogrammetrically captured and digitized. 3D artists worked out the details of the textures and cloths to recreate and animate the physical appearance of the puppets surfaces. In addition, Motion Capture recordings were used to add more puppet-like movement to the avatars. Large Language Models (LLMs) and generative AI form the basis of the interaction and conversations. In order to create a character prompt for each puppet, the original audio recordings of the pieces in which the specific puppets appeared, were transcribed and analyzed. The result of the analysis were distillations of the most characteristic words and/or expressions of the puppets, which were then inserted in the prompt. In addition, to the character references the prompts were further developed to fit the role of each puppets within the exhibition and its dramaturgy. Further, the interaction is supported by different systems, aiming at a flawless interaction and conversation. SAR motion, a system internally developed by Florian Bruggisser, detects via a camera the movement and body of a visitor. This triggers the interaction and the puppet starts to talk.

My role: Unity Development, Programming and Integration, User Experience Design, LLM Integration.