soundbite is an app that creates audio-reactive visuals based on audio tracks uploaded by the user. The animation is driven by Apple's Metal shader framework, and audio processing is done using AVFoundation and Accelerate vDSP. Though functionality is still being added, it is built for artists to create enticing visuals that serve as a more engaging alternative to voice memo screen recordings for teasing music to their audience.
AVFoundations
SwiftData
vDSP
Voice memo screen recordings are functional, but boring. Building custom animations or filming shots for a teaser is overkill for small artists.
soundbite is instant and effortless, generating audio-reactive visuals that make artist demos look as good as they sound.
Turn your sound on and press play below to see it in action.
Key components
Audio Upload: Upload songs from videos in your camera roll
Audio Analysis: Real-time FFT (Fast Fourier Transform) analysis using AVFoundation and vDSP
Dynamic Shaders: Custom Metal shaders that react in real-time to frequency data.
Customization: Granular control over primary/secondary colors and shader styles.
Offline Rendering: A dedicated offline rendering pipeline to export 60fps video.
Audio pipeline
When a track is imported, it is saved locally via FileManager, with metadata references stored in SwiftData.
When the editing canvas is active, Soundbite utilizes AVAudioEngine for playback. Then, the installTap() method is used to tap into the live audio feed, passing that data to a helper class. This class uses vDSP to perform a Fast Fourier Transform (FFT), normalizing the raw data and binning it into 64 distinct frequency bands. This array of "loudness" values drives the UI visualization.
Dynamic shaders
The animated backgrounds are powered by Metal shader math. To bridge Metal and SwiftUI, the [[stitchable]] attribute is used in the Metal function definitions.
This allows SwiftUI to recognize the shaders and apply them using the .colorEffect() view modifier. The normalized frequency data from the audio pipeline is passed directly into the shader, modulating specific visual elements (like position or color shift) based on the music's intensity at certain frequencies.
Next for soundbite
Node-Based Render Engine: Create a render engine that is Node based, which will offer near unlimited flexibility for rendering reactive FX. Map a traditional NLE on top of the node based architecture to make it usable without directly exposing the graph