Store Everything in Plain Text Files

Wishlist Draft wishes
1

What is the problem?

As a user I’d like all artifacts I create on my Synth to be in a format that is readable by humans and machines, so that I can inspect them, compare them, change them in a text editor, and easily write software that automates changes that are tedious to make on the device itself.

Closed and proprietary file formats inevitably prevents artists accessing their own work in the long run. There’s no need to do that, so we shouldn’t.

What should this feature achieve?

JSON is probably the simplest format, most modern languages have a JSON library for serializing and deserializing objects, because JSON is the standard for REST APIs.

Other plain text formats for structured data exist, but are less common (e.g. YAML), or even a general pain in the ass (XML).

All files on the SD card should be stored as plain text, e.g. chords, presets, scenes (including sequences), and also grid layouts. Here’s some examples what is possible then:

  • converting sequences from MIDI files, or converting sequences to MIDI files,
  • various cleanup and housekeeping tasks, e.g. renaming scenes according to their configuration of engines and polyphony
  • using a diff tool to see differences between a preset saved inside a scene, and an engine preset
  • assigning the same effects settings to multiple scenes (for live setups)
  • identify certain issues (e.g. totally messed up gain staging), and potentially fix them automatically
  • convert sounds for use with other synthesizers
  • create new grids

And most of that is even possible when one’s Synth is broken, or has been sold long ago.

Are there any workarounds?

Maybe: Reverse engineering the file format after each new firmware release. Ain’t nobody got time for that, tho.

Any links to related discussions?

This article makes a good case for "“if you can’t open it, it’s not yours.”. Mostly related to DRM, but a closed format has the same effect.

Any references to other products?

The list of synthesizers that have a well documented sysex format is substantial. Sysex is not plain text, but when it is documented it is an “open” format that allows for a multitude of editors and librarians to exist.

Just two examples spanning more than 3 decades

Here is a small selection of devices and applications that use plain text for storage (various formats, not just JSON, and not always for all things they store):

6 Likes
2

This might also be going in a similar direction but for the trackers:

1 Like
3

I agree with @paaat , this very much goes into the same direction as my wish.

I’ll check if we can generalize this across devices for a more open ecosystem.

2 Likes
5

The other wish contains 2 wishes, so I think there might even be 3 slightly different things in discussion:

  • supporting existing file formats for entire musical pieces (mod files, bitwig open format), I see that is already (partially?) solved
  • documentation for file formats
  • storing everything in plan text files files in a format that would not require much additional documentation besides the actual product manual to be useful to people

My experience with documentation for file formats is, when there are a lot of new features (which we all hope for), it will take an effort to maintain, and this will be neglected when there is pressure.

A plain text format with keys and values like JSON does not have that problem, you can save a new scene and diff it to a previous backup to see what is new.

So it does look similar, and might lead to a similar end, but there are substantially different paradigms at play. There are costs and benefits to each, and I think a choice needs to be made between them eventually. We should keep that in mind when generalizing across devices.

1 Like
6

You are right, it does not explicitly say that we’d like to see plain text files, but in essence we want the same thing.

Accessible open and documented files, so that we can build tools around them.

7

True. All I’m saying is that selecting the right format might reduce the amount of documentation that is required to “nothing”.

1 Like
8

That is very true!

1 Like
10

This is a fantastic idea, although I hate JSON with the fury of a thousand dying stars. Even though it is definitely less common than JSON or YAML, I’d propose TOML as it is more “human friendly” than either JSON or YAML. But any solution that is plain text would be wonderful.

11

Wait until you hear about XML. :wink:

But it’s so simple to convert to and from TOML, there’s no reason to touch a JSON file if you don’t want to.

12

I would have thought XML would be most user friendly for those with less tech knowledge - but then I have spent a lot of time modifying presets for the Artiphon Chorda and Orba 2. If you have done a little html XML is an easy adjustment.

13

Wait until you hear about XML. :wink:

I’ve been a software engineer for more than 20 years. I’m very familiar with XML. :sob:

14

Proprietary (binary) formats are also far more compact and much faster to read and write since the overhead of all that read parsing and write translation are unnecessary.

You don’t get anything for free.

I’m not saying the Synth doesn’t have the horsepower for this, but there are tradeoffs.

16

Proprietary and binary is not the same.

I would agree that binary formats can be more compact and faster, however but I would argue that this kind of concern is unwarranted in a device that renders 8 voices of realtime audio with effects, displays live waveforms and comes with a 16 B SD card, and where the number of files per directory is limited by the firmware to 250.

Specifically we’re talking about a tiny amount of data, patches are between ~200 bytes (FX) to ~800 bytes (sounds) here, and references to macros and samples are already stored as plain text in these files anyway, so the overhead of serializing and deserializing that is negligible. iI would also argue that sequences will actually be smaller as plain text, because at the moment even an empty sequence takes up 2k on disk.

In my experience as a developer, proprietary file formats have zero advantages, but lots of disadvantages. And as a user, I hate that content I create is locked away in proprietary formats especially when it is actually less effort to write open formats from the start.

I figured as much. I was inspired by your very poetic depiction of your relationship to JSON, and hoped to trigger something similar about XML.

@Sandroid I wanted to edit a typo in my previous post, but accidentally deleted it. Normally there is a button to get that back in discourse, but that is not visible here. I had to pull the text out of the editing history and post it again. Is that a a deliberate choice, or can that be fixed somehow?

1 Like
17

Indeed all propriety formats aren’t binary. I should have been more specific. Many are, as the point of using a compact, propriety format is often storage and speed optimization. Sometimes it’s also about purposeful obfuscation.

In any case, I agree open formats have many advantages. My point was that they aren’t free and on embedded systems like this they can have an impact on realtime performance. I’ve spent decades developing on small-scale systems like this as well.

If the file parsing only needs to be done on initial read and when written out during a Save operation then there should be almost no noticeable affect on performance from the user perspective. As you note, these are tiny files.

However, if there is any file access going on in realtime, which I think may be the case with the Play and Play+, then parsing can have a noticeable impact. In a case like this, serializing to/from a compact binary file can indeed have a significant performance advantage. The Play series apparently already have issues with complex projects taking a long time to save. Translating back and forth to a human-readable format might well make this problem worse.

Like you I would love to have access to the files for tweaking and just to understand things like chord fill templates. I have my doubts this will happen and I’m not sure all the Polyend products could do this without noticeable impacts.

18

I hear you. And yet my trusty old Sequentix Cirklon with a much less capable CPU and older storage media than any of the Polyend devices mentioned here stores all sequencer data and instrument definitions as JSON files, even with a lot of songs this happens smoothly in the background while it’s playing. So I guess there’s still hope.

What I also find encouraging is there are some examples where Polyend supports non-proprietary formats: Tracker apparently reads and writes .it and .mod formats.

19

Here’s hoping they see value in this request.

1 Like