PixelAudio is a Processing Library that provides various classes for moving between 2D arrays of pixel data (bitmaps) and 1D arrays of floating point values (audio signals) in indexing orders determined by space-filling curves such as a zigzag pattern, a Hilbert curve, and a Moore curve. You can turn a 2D image into an audio signal or turn a 1D signal (including live or recorded audio) into a 2D image. I created PixelAudio specifically to aid in a project that makes extensive use of Hilbert and Moore curves for creating a sort of color organ where sine waves determine the mix of pixel values (i.e., colors).
A pre-release version 0.8.1-beta, is now available for download from this repository. To run PixelAudio you will also need to install the Minim Audio Library. You can download it or install it from the Processing Sketch->Import Library... menu command. This version is only suggested for early testing. It is functional but incomplete.
In PixelAudio classes, 1D signals and 2D bitmaps are related to each other through lookup tables (LUTs) that map locations in the signal and bitmap arrays onto one another. You could think of the signal tracing a path over the bitmap, visiting every pixel. The signalToImageLUTin PixelAudioMapper lists the position index in the bitmap of each pixel the signal visits. Similarly, the imageToSIgnalLUT tells you what position in the signal corresponds to a particular pixel. This makes it easy to click on the bitmap and play an audio sample corresponding exactly to the location you clicked, or to transcode an audio signal into RGB pixel values and display them in a bitmap.
The PixelAudioMapper class and the PixelMapGen class and its subclasses provide the core functionality of the library and are abundantly commented. PixelMapGen provides a lightweight framework for creating mappings between audio sample and pixel data arrays. PixelMapGen subclasses plug in to PixelAudioMapper, which can read and write pixel and audio data while remaining independent of the mappings and the actual audio and image formats. It should be relatively easy to write your own PixelMapGen child class and have it immediately available to play with through PixelAudioMapper's methods. PixelAudioMapper also provides many static methods for working with audio and pixel arrays. Other notable classes include the WaveSynth class, which uses WaveData objects for additive audio synthesis to create both a playable audio signal and an animated image that are generated in parallel. Some of the coding examples show how you can read and write JSON files of WaveSynth configurations. Other animation and audio classes are in the works.
The examples currently provide a survey of features in the PixelAudio library, particularly for mapping audio signals and bitmaps, using JSON files for WaveSynth and PixelMapGen settings, capturing live audio, playing audio samples, and mixing color channels. There are whole sections yet to be added, but there's enough to get started, if you are curious. See the Examples README for descriptions of each example.
At this point (August 1, 2024) PixelAudio is at the alpha testing stage, functional but incomplete (so not yet beta). You can download it as a Processing library and run the examples and expect them to do interesting things. You will need the minim audio library, which can be installed with Processing's Import Library... command. I plan to release the library November 9, 2024, at Experimental Sound Studio in Chicago, where I am currently an artist in residence.
The PixelAudio library replaces an earlier library, PixelScanner, that was never released in a public version. It can still be found on my GitHub pages, with some of the code in PixelAudio included. Many of PixelScanner's classes have vanished from PixelAudio, replaced by a better scheme for class functionality with far less duplication of code. Here's a sample video created with the previous PixelScanner library (https://vimeo.com/767814419), part of my project for 150 Media Stream. An early demo of features for PixelAudio can be found here: (https://vimeo.com/880911064).