Technical set-up

The object-based interface engages the senses and activates the body.
To achieve this, a thorough investigation into basic electronics was conducted. Self-crafted objects are equipped with sensors to control
a digital editorial design process.

arduino & esp32

Initially, off-the-shelf buttons or potentiometers were connected to an Arduino Uno and a standard Firmata was used to control and trigger events in vvvv. Firmata is a general protocol that facilitates communication between microcontrollers and any type of computer software. This eliminates the need to write your own Arduino program.

Later, several DF Robot FireBeetle ESP32-E microcontrollers, powered by a LiPo battery, were used to make the setup wireless. Designed specifically for IoT, they support Wi-Fi and Bluetooth communication. Its „small size, ultra-low power consumption, integrated charging circuit and easy-to-use interface make it convenient for smart home IoT, industrial IoT applications and wearable devices.“

The open-source Arduino platform serves as the link. The ESP32 Dev Module is selected in the Arduino IDE (Integrated Development Environment) and the desired code for reading the connected sensors is loaded. A separate sketch is read into the ESP32 for each object and each individual sensor. The code remains clear as they are all simple analogue or digital inputs. Only information about the connected pins is required. The code for establishing a connection between the ESP32 and the MQTT broker and for transmitting the values to the latter was somewhat more complex, but once created, it could be used for all of them.

mqtt

A private Wi-Fi network was established utilising a TP-Link router, thereby enabling the placement of objects and unrestricted movement without the necessity for cables. Furthermore, a local MQTT environment was configured using the Eclipse Mosquitto MQTT broker and a Raspberry Pi. The Raspberry Pi is a compact single-board computer connected to the router via a LAN connection.

MQTT (Message Queuing Telemetry Transport) is a lightweight publish-subscribe messaging protocol commonly used for IoT applications. In this configuration, the Raspberry Pi, which is assigned a fixed IP address, acts as an MQTT broker. Devices (clients) connect to the same Wi-Fi network and to this broker to publish and subscribe to messages categorised by topic. Multiple devices can subscribe or publish to the same topic or to multiple topics. The broker, Mosquitto, is responsible for managing these topics and ensuring that messages are delivered to the intended subscribers. In this setup, several ESP32 microcontrollers are configured using the Arduino IDE and publish data to the broker under specific topics, which can be nested. The topics begin with "InShape" and are then specified by the object (e.g. "InShape/Jump"). The data can be read in vvvv using this string. This configuration permits the utilisation of objects as intuitive input devices, thereby facilitating the seamless integration of technology into their operational processes.

Additionally, the router is linked to the Internet for the Microsoft Azure speech-to-text application utilized in vvvv, though this is not required for the MQTT network itself.

sensors

Handcrafted textile sensors or standard readymade sensors are incorporated in the objects. An understanding of the fundamentals of electronics was achieved through testing and research. It is pertinent to note that no prior knowledge existed to this point.

A sensor is an electronic device that is capable of detecting and identifying changes in its surrounding environment. The sensors employed in this project use either DigitalWrite or AnalogWrite. These are functions to output signals on the selected port. DigitalWrite will set the output to one of two states, LOW or HIGH (equivalent to a value of 0 or 1). The AnalogWrite of thee ESP32 writes an analog value between 0 and 4095.

Handmade soft sensors are ideal for tactile interactions as well as for hiding the technical part and integrating it into the objects. The different textile techniques can make the hands and body move in different ways. They respond to a variety of movements such as stretching, pushing or squeezing. KOBAKANT's work served as a guide. The work of artist Michelle Vossen was also an inspiration, as she embeds handmade sensors in garments, creating textiles as interfaces.

During the investigation of various sensors, it was found that the use of ready-made sensors sometimes leads to undesirable reactions with objects of a certain size. This was observed, for example, with the "Long Flex Sensor" from Adafruit, which recognises a deflection in one direction, but shows a reaction even with slight inclinations. This would have made it difficult to find a suitable treshold when the sensor would have been installed in a larger object.

Tilt sensors, known as tilt ball switches, were also evaluated. These sensors detect basic motion or orientation. When the metal tube is elevated to a vertical position, a small metal ball, which rotates within it, rolls onto the contacts that extend from the end and establishes an electrical connection between them (Adafruit Industries, o. J.). However, the outcome was once again unsatisfactory, as the triggering resembled an uncontrollable shake sensor. This made it challenging to incorporate it into a larger object that responds sensitively to movement. The utilisation of small vibration sensors in medium or hard strengths did not yield a satisfactory outcome. They were only triggered by a precise flick of the finger, resulting in the sensors breaking at the feet after some time.

Tests were also carried out with stretch sensors. Firstly, attempts were made to realise stretch sensors with conductive thread or fabric. Unfortunately, the tests did not yield any functioning sensors with no prospect of scalability. The Conductive Rubber Cord Stretch Sensor from Adafruit, made from rubber impregnated with carbon black, was also tested. The resistance increases under tensile load. However, the values of this sensor were very imprecise and it tore quickly. KOBAKANT is researching various approaches to manufacturing knit stretch sensors. With more time it could have been the right approach to built a larger stretch sensor suitable for the project. Applications have already been implemented for the software that could be used optimally to influence the kerning of the font or change the image size.

Additionally, self-made, pressure-sensitive resistors constructed from conductive PE foam were subjected to testing. The varying thicknesses of the resistors were found to influence the strength of the pressure applied. These resistors demonstrated excellent performance. Nevertheless, the implementation of conductive fabric proved to be a more straightforward and scalable solution.

At some point, researching sensors and building DIY sensors became quite time-consuming. It is a field of research in itself. However, it was an important part of the design research for this project, as it was essential for the imagination to realise the objects. It was only with a growing understanding of electronics that suitable sensors could be found and developed to adequately realise and move the objects. This also had a strong influence on the aesthetic result.

vvvv

Instead of using the industry-standard desktop publishing software, InDesign, a customised application was coded. The designed software focuses on typical desktop publishing tasks and is reduced to simple actions such as inserting text and images.

The visual programming language vvvv, which facilitates rapid prototyping, was employed to develop the creative design software. This enables the static poster to be designed dynamically through interaction. The physical objects that serve as interfaces communicate with the program written in vvvv, allowing active interaction with the digital content.

As vvvv is only compatible with the Windows operating system, it was necessary to create a partition for Windows and install Bootcamp before proceeding.

vvvv consists of ready-made programme modules that can be called up from an extensive library and combined in a variety of ways. Each has a distinct purpose and has the ability to receive, process, and output data. These nodes are linked together via pins to combine functionalities and form a program. vvvv is a stream-oriented programming language that runs from the top to the bottom node of a patch. Because vvvv runs in real time, values can be changed directly and the result is immediately visible. vvvv has a frame rate of 60 fps, which means that the programme is executed 60 times per second. In each frame it is executed exactly once from top to bottom (Barth et al., 2013).

In addition, vvvv has many protocols for communicating with peripheral devices, making it perfect for physical computing projects (Barth et al., 2013). Although there are not yet many tutorials on the programme, vvvv has an active and helpful community that responds quickly to questions in the forum and helps to solve problems.

During the development stage of the program "InShape," hardware values were simulated using Boolean nodes. Booleans represent conditions and can be used to switch between several functions or values. For instance, a toggle can be turned on and off by right-clicking on it. If it is inactive, it sends 0, while if it is active, it sends 1. A bang transmits the value 1 for one frame when it is triggered. Furthermore, a press Boolean transmits the value 1 as long as it is pressed (Barth et al., 2013). Subsequent to this, the data received from the microcontrollers via MQTT were linked to different Booleans in vvvv in order to trigger the desired functions.

Due to the short learning span and the resulting lack of experience, repetition, randomness and rhythm played an important role in the creative process. However, this approach is consistent with the concept of allowing chance and flexibility in the design process.