Adobe Firefly Sound Generator Debuts – Custom SFX Guided by Your Voice

Adobe continues to work on generative AI and has just rolled out a web-based sound generator in beta, powered by the company’s neural engine, Firefly. It allows users to generate custom sound effects from text descriptions, but that’s not the most interesting part. You can also guide the model with your voice, recording audio cues to precisely time sound effects with on-screen actions. How exactly does the new Adobe Firefly sound generator work, and where to try it out – read in our article below!

Generating sounds from text is not a new concept. Several companies have been offering this feature for a while now (with ElevenLabs at the forefront, for example). However, the idea of voicing your desired effects so that AI knows precisely the length and the energy of the action in the video is exciting. It offers much more control and inspiration for creative results. Think of it like Foley: you record something fast with just your voice, and it gets turned into the proper sounds.

Adobe Firefly sound generator and what it offers

The new tool by Adobe looks more like a compact browser-based editor. It allows users to upload video input (up to 30 seconds in length at the moment) and then generate sound effects for it, aligning them directly to the correct time stamps.

Adobe Firefly’s sound generator combines a text prompt with your vocal performance to produce a sound effect that closely aligns with the description while matching the timing and dynamics of your voice. The tool also allows users to create multiple audio tracks and layer different sound effects for a richer soundscape.

Please note that the recorded voice notes are only used to guide the sounds. This Adobe tool does not generate speech or sung lyrics.

How can I use the Adobe Firefly sound generator?

Actually, the web interface of the new sound generator is quite intuitive to use. First, go to Firefly’s homepage, find the Audio module, and select “Voice to sound effects (beta)”. You will also see one with “Text to sound effects (beta)”, but you won’t really need it. Providing recorded guidance is optional in the first tool as well, in case you prefer to work with prompts only.

The next steps:

• Upload the media you want to generate sound effects for. (It could be an audio or video file.)
• Move the playhead on the timeline to the exact moment where the first sound effect should begin. (You can adjust it later, but setting it accurately now will help with subsequent recordings.)
• Now go to the Voice performance section on your left. There, you can either upload an audio file from your device or record directly in the interface. If you click on the latter function, a three-second countdown will appear. After it’s finished, you can start recording.
• You will see that your voice notes were added in the prompt tab above. Type in your text description there. We’ll share some tips for prompts in a moment.

Once you click “Generate,” the AI will come up with four sound effect variations. Select one, and the program will insert it at the playhead’s position. If you want to exchange it, click on the actual clip in the timeline first, and then choose another generated version.

If you need to layer multiple effects, click on “Add Audio Track,” and repeat the process from the beginning. Apart from that, it’s recommended to work with one sound at a time.

Trying out the new tool

We decided to give it a try and uploaded some snippets of Johnnie cooking from this special episode of weekly CineD’s podcast “Focus Check” on healthy fuel for filmmakers. I edited the clips into a single video and uploaded it, then worked on each sound in Firefly individually.

Here is how the video looks with just my raw audio hints. I recorded them on my smartphone while traveling by train, hence the noisy surroundings (which didn’t seem to be a problem for Firefly’s processing at all):

After adding simple text prompts to each of the sounds, like “a knife cuts radish on a wooden cutting board” or “a kitchen blender mixes” I placed the generated sounds on the timeline, and tweaked their volume a bit (which you can do by clicking on each of the created clips separately). And this is my quick result:

The timing and quality of the generated sounds are indeed quite impressive. Even more so, as this whole process took me roughly 10 minutes!

After you finish working on your project, you can either download the final video file or the generated SFX as separate files to work on them further.

Some tips for writing prompts

Developers of Adobe Firefly sound generator also shared some tips for writing prompts that will get you closer to the desired results. For instance, it’s important to provide a clear, concise, and direct description of the sound you want. For example, don’t write “the sound of a lion roaring.” Instead, keep it at “lion roaring.”

Another tip is to use adjectives and verbs. The former should describe the qualities of the SFX, and the latter – convey the action or behavior of the sound. This approach allows for more precise control over the characteristics of each sound effect.

For more prompting tips, head over here.

Price and availability

Adobe Firefly sound generator is now available in beta as a web-based application. Firefly offers various pricing plans, described here. And if you are a Creative Cloud user, then you can try out Adobe’s generative AI, including all the latest features, as part of your subscription straight away.

What do you think about Adobe Firefly sound generator? What future applications do you see for it? In which projects would you use it? Let us know in the comments below!

Feature image: Adobe