sdf PhD

overview

links

portfolio

thesis

Exploring visual representation of sound
in computer music software through
programming and composition

thesis — §1 — §2 — §3 — §4 — §5 — §6 → §7 ← toc — bib

— §7.1 — §7.2 — §7.3 — §7.4 — §7.5 — §7.6 —

Selected content from
a thesis submitted with a portfolio of works to the University of Huddersfield in partial fulfilment of the requirements for the degree of Doctor of Philosophy

December 2013

Samuel David Freeman

Minor amendments
April–June 2014

7: Conclusion

7.1 Framing the project

My objective has been to document the creative process of developing an approach to computer musicking that is both broadly aware of its technological heritage, and specifically focused on the visual interface between human mind and soundmaking DSP algorithms.

This thesis has described and contextualised my compositional responses to questions about representing aspects of sound visually in interactive systems. The project began by asking the following primary research question which has been found to relate strongly to the concept of technê: How might the ways in which sound is visually represented influence the choices that are made while those representations are being manipulated and organised as music? Additional questions, indicated at the initial stages of the project (§1), were: Which aspects of sound are represented visually, and how are those aspects shown?

Taking those three questions as the starting point for investigation and compositional programming towards new works of art, the project did not seek direct answers in a quantifiable sense, but instead pursued a techno-aesthetic dialogue which eventually took forms that could not have been predicted at the outset.

Already established at the start of this project were the bases of the spiroid-frequency-space concept, which had emerged through my practice of sketching on paper in the process of composition, and the scripted drawing system called thisis, which I had been developing to assist in that practice. Having grown weary of rectilinear representations and overt skeumorphism in computer music software I looked to the circular form of the spiroid, and the abstract geometry of thisis, as foundations to an alternative paradigm.

7.2 Contextual grounding

Recognising sound as a psychophysical phenomenon, the aim of §2 was to identify which fundamental aspects of sound were important to me in the context of music. Looking at how physical aspects, and abstracted representations psychological aspects, of sound can manifest visually, the following themes were highlighted: the sense of ever-present 'now' of Cymatics (in which one can see patterns being made whilst sound is happening); the concept of a recording sound as a temporal trace (in which one can see the patterns made after sound has happened); that the harmonograph is interesting (tracing harmonic interactions at a rate comprehensible to the human eye); and that pitch is a morphophoric medium, as is visual space (each being a medium in which forms retain identity through transformations), but colour is not a morphophoric medium (and herein lies the basis for what I call the chroma paradox).

Many of the works in the portfolio are conceptually grounded by aspects of live coding. As the project progressed this contemporary research-field was found to have wider reaching significance in understanding software as the medium of computer music than may have been anticipated at the outset.

7.3 Techno-aesthetic development

The six study works that are described in §3 seek in different ways to understand better what visual representations in software are, and how they are being used. Starting from the recognition that the computer screen is but a matrix of pixels, works such as visyn, sub synth amp map, and the two gramophone model examples are all based on the manipulation of data-matrix cell values that are taken for display on screen. The connections in these works between aspects of sound and the data-matrices displayed on screen vary:

• for visyn the synthesised visual forms are abstract, while sound is made from the data of the visual display (limited success with this approach, but there is potential for further work if more computationally efficient methods were applied);

• in sub synth amp map the amplitudes of (any) two digital audio signals are mapped to spatial coordinates that create temporal traces on screen for direct visualisation of sound as motion (think Lissajous, lamp light, laser beams, and laptop screens);

• the gramophone model encompasses two types of data-matrix display:

  • the temporal trace type of display (as in sub synth amp map) is used again, in this case to show where a modelled stylus is now or has just been in the space; and

  • the other way that data-matrix cells are used is as memory for audio sample data-values in the modelling of the gramophone record surface (arranging a stream of sample values as a spiral path on the plane);

Of these three approaches to linking the screen and sound it was with aspects of the gramophone model that I found the most creative promise. Having developed the gramophone model as a speculative curiosity, its key features would later become an integral part of sdfsys as the tildegraph concept.

7.4 Tildegraphing

Whereas it is sound input to sub synth amp map that makes patterns on screen,^[n7.1] the tildegraph concept, that eventually emerged during the project, allows parametric control of the visual patterns on screen which can then interact to make sound. This reversal of roles (sound before screen / screen before sound) was indicated as an objective, quite early in the project, in relation to Cymatics (see end of §2.1.1). In further work to extend that idea, one could implement a hybrid system to explore the interaction of those roles by using the sdf.sys.dv module with a digital video camera focussed on a Cymatic setup in order to take patterns generated by sound (in the physical world) into (the virtual world of) sdfsys; one could then use a ~g.r. module to make sound by reading data from the video matrix. Such a configuration would be similar the to Tüb installation (Erlach et al., 2011) described at §(6.2.6.c). In the hypothesised configuration a feedback loop between the physical world and sdfsys via audio and visual domains could be created by letting the sound input to the Cymatic system be taken from the output of the tildegraphing in sdfsys.

[n7.1]   A system that promotes close consideration of the primary research question because the visual manifestations in that piece certainly affect the choices that are made while controlling the input sound.

Exploration of tildegraph concept in sdfsys, as a synthesis technique founded on techno-aesthetic concerns, presents fertile ground for further research, both within sdfsys and with the potential for alternative implementations. The tildegraph concept thus stands as a significant contribution of this practice-based research.

7.5 Music via visual representations

In the act of controlling tildegraph modules in sdfsys there is a negotiation of cognitive balance between targeting the visual and the audio consequences of parametric input; this leads to a particular sense of audio-visual synergy that is (seemingly) unique to the sdfsys environment.

Many of the parameters used in tildegraph modules for sdfsys are to set frequencies within their spatiotemporal algorithms, and a spiroid-frequency-space mapping is used to translate the location of a Hz type sprite in the main area of the sdfsys gl-ui to a frequency value.

The spiroid concept has its own chapter (§4) in which its genesis within my own practice is explained before the concept is contextualised as an archetypical form of visual representation. It is noted that although spiroid-like spirals are common in print, they are rare on screen; or at least they were: recent and forthcoming audio analysis softwares have been cited that use what I call the spiroid-frequency-space. In future work I hope to promote the use of this concept for input (of parameter values) as well as output (of spectral analyses on screen). My work on the spiroid concept within this project contributes both in the provision of a collated contextualisation, and also in the giving of that specific name which was formed after two observations: one, that a perfectly good word (spiroid) seemed not to be used much in contemporary language, and then also that there is a profoundly beautiful concept in the visual representation of sound that was both related to that word, and also not being used as much as perhaps it could be.^[n7.2]

[n7.2]   Though that description puts the cart before the horse, somewhat, by putting spiroid before the frequency-space.

Pursuing other circular representations to complement the spiroid-frequency-space in software for new musicking, and employing the thisis system for drawing, I developed the CirSeq time-space; this, and my explorations with it, are described in §5. Some elements of the CirSeq development that were not covered in this commentary include the work on cirseq_beta (in the form of unit modules for sdfsys_beta), and the exploration (on paper) of a triangular formation following the same harmonic series of systematic quantization maintaing angle-to-phase and radius-to-temporal-resolution correlations. Inquiries related to CirSeq could benefit further research into new interfaces for musical expression.

The saw~onepole~noise~click~ is a composition that connects to many of the themes that have emerged during this project whilst, of itself, taking a different approach to visual representation of sound. Bringing attention to the visual medium of source code in software, saw~onepole~noise~click~ is composed in the form of a single maxpat such that nothing is hidden, as shown in Figure 3.37. The visual representation of soundmaking processes in the maxpat source code for this piece is both complete and human readable: the prescribed timing, timbre, and temporal processing of sonic events are all presented in full technical detail, including those aspects of the piece that are indeterminate. In this case the software is most definitely a score for the piece of music, but it is also the instrument used for, and the player of, that score.

As well as being derivative of a live coding exercise, saw~onepole~noise~click~ is contextualised as an aesthetic exploration of the dichotomy of time-domain and frequency-domain thinking which is a common conceptual basis in computer music. Discussion both of saw~onepole~noise~click~ (§3.6), and of CirSeq Zero (§5.3–§5.4) have shown that the distinction between those two domains is a matter of perspective and choice. The CirSeq (time-space) concept was made to be used in tandem with the spiroid (frequency-space) concept for controlling parameters of sound and making music, but I have subsequently found that the spiroid can be enough on its own to control both time- and frequency-domain aspects of soundmaking as durational elements of form.

7.6 The sdfsys environment

It is always 'now' when one is working in the sdfsys environment; there is no 'time-line' representation – unless one chooses to classify the spiroid under that term, but spiroid curve is a line that represents a continuum of durations, rather than representing the continuation of time within a duration – there is also no undo feature in sdfsys, and very limited save and recall functionality. It can thus be described as a live coding environment that has been explored compositionally. Different approaches to composing with sdfsys have been described (§6.4) in the presentation of orbit8 reading cyclephase, febandflow, five four three, penhexsepoct, and more cogmos, and many more approaches as possible.

With the tildegraph concept in sdfsys, one is able (for example) to use a combination of near-miss harmonic intervals when specifying frequencies to establish a slowly shifting timbral soundscape that may take minutes to evolve without requiring any further input control. The human is then free to enjoy the audiovisual experience with the option to affect further control over the soundmaking processes. There is also the dimension of geometrical drawing via thisis commands which may or may not also become part of the soundmaking, and promotes full-minded engagement with the present moment in a temporal process of creating form. To my knowledge, this workflow is a unique contribution to the art of computer music.

The sdfsys environment is an open ended techno-aesthetic entity that brings together a wide range of historically aware computer music concepts and techniques.