A Postmortem on VoiceFX

With the PR post out of the way I'd like to give a more personal update on why I (Xaymar) have made this decision so this will be more of a history thing of the entire thing.

I started this project back when NVIDIA announced their machine learning based noise removal in 2020 and at the time NVIDIA hadn't even publicly announced the SDK for it. I didn't know there was going to be an SDK and worked with a reverse engineered header for the whole thing. And it seems NVIDIA didn't know that I didn't know because their legal team had accidentally leaked this information to me when I announced VoiceFX.

As I had zero experience in making VST plugins (or plug-ins as Steinberg calls them) at the time my focus shifted towards that for the time. Finding a VST 3 SDK was easy as it was publicly available under a mostly agreeable license but I had no such luck with VST 2 SDKs. Almost all of them were based on stolen/leaked code or even just the stolen/leaked code. So I did what any definitely (in)sane person would do and spent a bit over 15000 € on hiring coders and lawyers to document the stolen/leaked code and then reimplemented it from the documentation under the watchful eye of a lawyer.

This marked the creation of the first ever clean room VST 2 SDK in existence with no legal licensing issues or stolen/leaked code and the accrued debt for the whole thing was paid off in March 2024 with the combined income from StreamFX and VoiceFX. It is now in use by projects such as DPF, RS-MET, YADAW, and as you might have guessed already: VoiceFX. I'd like to offer a "Thank you!" and a virtual hug to everyone who purchased the project or supported me via Patreon at the time and helped me pay off the debt.

This SDK has given me a quite deep insight into why audio engineers tend to have no to few ideas why something is broken. Quite frankly the VST API is a horrible base for audio plugins but it is one of the bigger standards for years now. Despite that I've managed to rapidly improve the project and extinguish fires as they appeared to the best of my ability. Unfortunately with the ever increasing cost of living due to various factors outside of my control I had to shift my focus towards projects that actually brought in a livable income.

If someone wishes to continue VoiceFX they have my blessing to do so. It's missing a few things but is otherwise in a functional and workable state. The only thing I'd throw out is VSTGUI which would be replaced with SDL2 and IMGUI instead. VSTGUI is an absolute nightmare to work with and I don't know how anyone could reasonably use it. Anyway, I hope this explains the whole situation. I'm just one person and I'm not exactly excited for burnout after having gone through 11 of them at the age of 31.

So now for the postmortem part: What went wrong?

• I shouldn't have created my own framework and instead used an existing one. It could have saved me months of debugging time and likely made everything work much faster.
• Relying on the documentation as what other implement is a terrible idea. Only Adobe Audition implemented what the documentation says. No other host out there implements what is documented at all and even the widely popular JUCE framework is guilty of this.
• Telling people that something isn't designed for real-time usage somehow doesn't prevent them from trying to use it for real-time usage anyway. Still haven't figured out why words don't work with some people unless you tell them the exact same thing directly.
• Many hosts do not understand that in order to support Windows 8/8.1/10/11 they actually need to either ship a manifest or embed it as a resource. Having to write Windows 7 compatible code in something that can only ever run on Windows 10 or 11 was interesting.
• A few hosts sandbox plugins to such a degree that basic Windows functionality isn't available in the process they sandbox the plugin in. I don't know why they do this as you can simply break out of that sandbox with a few direct calls to undocumented NT APIs.
• Running GPU work in the same thread as audio processing happens is probably not a great idea and I should have figured out how to offload processing to a thread early on. Maybe it would have even solved the whole real-time support problems.
• Couldn't ever get DaVinci Resolve to work because their VST sandboxing process is fucked up in weird ways.

What went great?

• The multi-windows-version ring buffer was quite nice to implement and learning how to abuse mapped memory has made Hellextractor much faster than it used to be. It's weird that when you map a file into memory you get much faster I/O throughput than if you were to read the file directly but it's a nice performance improvement.
• Reverse Engineering a whole SDK is probably not for me as a solo developer. It was a nice learning experience but I'd rather do that together with other people. Though it has given me much deeper insight into x86 and x86-64 assembly.
• Learning how to deal with NVIDIA's Maxine SDKs in a sane way has enabled me to write safer and faster GPU code. StreamFX has already received some of those upgrades which cut several hundred microseconds of render time.