Implement VST2 SysEx events

Apparently these _are_ actually used. Sometimes.

CHANGELOG.md

@@ -8,12 +8,21 @@ Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
 
+### Added
+
+- Added support for VST2 plugins sending and receiving SysEx events. Certain
+  MIDI controllers like the _Arturia MiniLab Mk II_ output SysEx events when
+  changing between octaves, and hosts like **REAPER** forwards these events
+  directly to the plugin. Before this change this might cause crashes with
+  plugins that try to handle SysEx events, like the _D16 Group_ plugins.
+
 ### Fixed
 
 - Fixed a regression from yabridge 3.5.1 where certain non-standard compliant
   VST3 plugins wouldn't resize to their correct size when opening the editor.
   This affects **Kontakt**, and it was caused by reverting just a little bit too
   much code in the regression fix from the previous release.
+- Fixed _D16 Group_ plugins crashing when the host tries to send SysEx events.
 
 ## [3.5.1] - 2021-06-31
 

README.md

@@ -556,9 +556,6 @@ There are also some extension features for both VST2.4 and VST3 that have not
 been implemented yet because I either haven't seen them used or because we don't
 have permission to do so yet. Examples of this are:
 
-- SysEx messages for VST2 plugins. In addition to MIDI, VST 2.4 also supports
-  SysEx. I don't know of any hosts or plugins that use this, but please let me
-  know if this is needed for something.
 - Vendor specific VST2.4 extensions (for instance, for
   [REAPER](https://www.reaper.fm/sdk/vst/vst_ext.php), though most of these
   extension functions will work out of the box without any modifications).

src/common/logging/vst2.cpp

@@ -428,7 +428,13 @@ void Vst2Logger::log_event(
                 },
                 [&](const AEffect&) { message << "<nullptr>"; },
                 [&](const DynamicVstEvents& events) {
-                    message << "<" << events.events.size() << " midi_events>";
+                    message << "<" << events.events.size() << " midi_events";
+                    if (!events.sysex_data.empty()) {
+                        message << ", including " << events.sysex_data.size()
+                                << " sysex_events>";
+                    } else {
+                        message << ">";
+                    }
                 },
                 [&](const DynamicSpeakerArrangement& speaker_arrangement) {
                     message << "<" << speaker_arrangement.speakers.size()

src/common/serialization/vst2.cpp

@@ -41,13 +41,28 @@ DynamicVstEvents::DynamicVstEvents(const VstEvents& c_events)
     // Copy from the C-style array into a vector for serialization
     for (int i = 0; i < c_events.numEvents; i++) {
         events[i] = *c_events.events[i];
+
+        // If we encounter a SysEx event, also store the payload data in an
+        // associative list (so we can potentially still avoid allocations)
+        const auto sysex_event =
+            reinterpret_cast<VstMidiSysExEvent*>(c_events.events[i]);
+        if (sysex_event->type == kVstSysExType) {
+            sysex_data.emplace_back(
+                i, std::string(sysex_event->sysexDump, sysex_event->byteSize));
+        }
     }
 }
 
 VstEvents& DynamicVstEvents::as_c_events() {
     // As explained in `vst_events_buffer`'s docstring we have to build the
     // `VstEvents` struct by hand on the heap since it's actually a dynamically
-    // sized object
+    // sized object. If we encountered any SysEx events, then we'll need to
+    // update the pointers in `events` to point to the correct data location.
+    for (const auto& [event_idx, data] : sysex_data) {
+        auto& sysex_event =
+            reinterpret_cast<VstMidiSysExEvent&>(events[event_idx]);
+        sysex_event.sysexDump = const_cast<char*>(data.data());
+    }
 
     // First we need to allocate enough memory for the entire object. The events
     // are stored as pointers to objects in the `events` vector that we sent

src/common/serialization/vst2.h

@@ -89,7 +89,9 @@ struct ChunkData {
  * A wrapper around `VstEvents` that stores the data in a vector instead of a
  * C-style array. Needed until bitsery supports C-style arrays
  * https://github.com/fraillt/bitsery/issues/28. An advantage of this approach
- * is that RAII will handle cleanup for us.
+ * is that RAII will handle cleanup for us. We'll handle both regular MIDI
+ * events as well as SysEx here. If we somehow encounter a different kind of
+ * event, we'll just treat it as regular MIDI and print a warning.
  *
  * Before serialization the events are read from a C-style array into a vector
  * using this class's constructor, and after deserializing the original struct
@@ -116,14 +118,32 @@ class alignas(16) DynamicVstEvents {
     /**
      * MIDI events are sent just before the audio processing call. Technically a
      * host can call `effProcessEvents()` multiple times, but in practice this
-     * of course doesn't happen.
+     * of course doesn't happen. In case the host or plugin sent SysEx data, we
+     * will need to update the `dumpBytes` field to point to the data stored in
+     * the `sysex_data` field before dumping everything to `vst_events_buffer`.
      */
     boost::container::small_vector<VstEvent, 64> events;
 
+    /**
+     * If the host or a plugin sends SysEx data, then we will store that data
+     * here. I've only seen this happen with the combination of an Arturia
+     * MiniLab keyboard, REAPER, and D16 Group plugins. We'll store this as an
+     * associative list of `(index, data)` pairs, where `index` corresponds to
+     * an event in `events`. There's no 'small_unordered_map' in
+     * Boost.Container, so this will have to do.
+     */
+    boost::container::small_vector<std::pair<native_size_t, std::string>, 8>
+        sysex_data;
+
     template <typename S>
     void serialize(S& s) {
         s.container(events, max_midi_events,
                     [](S& s, VstEvent& event) { s.container1b(event.dump); });
+        s.container(sysex_data, max_midi_events,
+                    [](S& s, std::pair<native_size_t, std::string>& pair) {
+                        s.value8b(pair.first);
+                        s.text1b(pair.second, max_buffer_size);
+                    });
     }
 
    private: