piper VS willow

Hello HN -

My goal is to have a high-quality open-source text-to-speech engine running on my Mac. Research has led me to evaluate Piper TTS (https://github.com/rhasspy/piper) - the voices sound good, and it can run on a RaspberryPi. There seems to be a blocker in getting it to run directly on a Mac (this person explains why: https://www.youtube.com/watch?v=hQe861JElXc).

So, I've been trying to get Piper to run in a Docker container on my Mac.

I've spent all evening trying to get this to work, and haven't succeeded. I can't seem to get the right incantation of Docker files & compose.yml to get it running.

Who's been able to get Piper running locally in a Docker container, and can you share guidance?

Here are my working notes for Piper & Docker: https://jonathan.rogivue.net/notes/piper-text-to-speech/

Thank you!

This is so great--both in terms of the project & the write-up. Thanks for sharing your work! :)

A "quick" dump of some thoughts it provoked:

(1) Really like the "meaning-nonsense continuum" concept--and that neither extreme is explicitly labelled in the image immediately following where the term is introduced. :)

(And, yes, "Gravity learns about regret." is indeed kind of beautiful.)

(Aside: Adding alt text to the generated images would be beneficial both for copy/pasting & accessibility reasons. :) )

(2) This statement made me smile: "If you don’t enjoy reading and contemplating these sentences like this, we are simply very different people." Because while I very much fall into the category of "enjoy reading and contemplating these sentences" I can also imagine people who definitely don't. :D

(3) Part of the reason for (2) was because I'd already encountered a couple of the "don't work" examples where my immediate thought was "wait, how about in this situation though?". e.g. "Aligning with compass."

Which then got me thinking about stylistic, metaphor or time-period related aspects of grammar, e.g. pirate treasure map instructions written in cryptic sentence fragments.

(4) Which leads into the whole "procedurally generated diagrams/documents" aspect of the project that I also think is particularly cool. It immediately got me thinking of being in a game and walking into an inventor character's lab or a magician's library and finding all manner of technical or arcane drawings generated for either environment or game mechanic purposes (e.g. in a detective/mystery game like "Shadows of Doubt").

Then again, since I was a kid I've always had an interest in browsing technical component and, err, office stationery catalogues, so maybe it's just me. :D

It also made me think back to "Interminal" an entry to the 2020 ProcJam procedural generation game jam, which generated an airport terminal including duty-free stores with procedurally generated perfumes (including "name, visual identity and smell"): https://nothke.itch.io/interminal

(5) The "physicalization" aspect of the project to render it in a tangible form is also cool--particularly because (at least to me but maybe also to all those vinyl-owning kids with no turntable :D ) a physical form seems to the amplify the "meaningfulness" of a message...

The underlying "digitalized cursive handwriting" project was also of interest--though I only skimmed its write-up in an attempt to avoid yet another rabbit hole. :)

The handwriting system seemed like it might also be amenable to animated use--which then made me think, "Wait! I've seen something about that recently...", had no idea what, but then some background retrieval process just produced the answer as I was writing, it was Noclip's recent "The Making of Pentiment" documentary: https://www.youtube.com/embed/ffIdgOBYwbc

(6) With regard to the technical execution/implementation side of things: I've observed that a project's need for text generation often raises a question of what implementation approach to use, in terms of a DIY vs pre-existing solution.

One issue which affects text generation specifically is "general solutions" often seem to tend toward specialization over time (e.g. "text generation for interactive fiction", "text generation for branching narrative-driven games", tools such as Yarn Spinner[0] & Ink[1]), which ends up making the solutions less suitable for simpler/different use cases & increases difficulty of the learning curve.

This was something I ran into during a small sub-project[2] last year where the text generation was somewhat "incidental" to overall project goal. I started out with a "quick DIY" solution but still ended up spending enough time on that aspect that I started to wonder if I'd be better not entirely re-inventing the wheel.

Around this time I ran into the "Blur Markup Language"[3][4][5] project which has the tag line "write text that changes" and--while I haven't yet used it--seems like it might be a promising "mid-level abstraction" solution for text generation, so thought I'd mention it as a potential option for others with text generation needs.

(7) In terms of other helpful text generation related resources, I've found various "word lists" to be of use, so thought I'd mention this "weasel words" list as a starting point: https://github.com/words/weasels/blob/main/data.txt#L14

(The repo README also links to other word lists under the same org including word categories such as "buzzwords", "filler", "hedges" & words listed in order of associated positive/negative sentiment.)

Thanks again for sharing your work & look forward to seeing where your projects go in future, should you share more in future. :)

---- footnotes ----

[0] https://github.com/YarnSpinnerTool/YarnSpinner

[1] https://github.com/inkle/ink

[2] I wanted to generate "scripted dialogue" samples[2a] to demonstrate the 900+ individual Text-To-Speech speaker voices in the Piper TTS[2b] LibriTTS voice model[2c], in a form that is: useful for evaluating the voices; not incredibly tedious to listen to; and, makes it possible to identify which speaker you are currently hearing.

[2a] Subset of resulting generated[2d] speech output can be heard in the second example here: https://rancidbacon.gitlab.io/piper-tts-demos/

[2b] https://github.com/rhasspy/piper

[2c] https://huggingface.co/rhasspy/piper-voices/blob/main/en/en_...

[2d] Text generation script: https://gitlab.com/RancidBacon/larynx-dialogue/-/blob/featur...

[3] https://bml-lang.org

[4] BML intro/overview: https://bml-lang.org/docs/guide/language-basics/

[5] Online BML editor with syntax cheat sheet: https://bml-lang.org/sandbox/

My interest in TTS is around "indie" game creation, animation and "radio plays".

A couple of years ago I started development of a tool to help with the generation of game audio such as NPC dialogue, "barks" or narration for those without access to/budget for human voice actors: https://rancidbacon.itch.io/dialogue-tool-for-larynx-text-to...

One thing I found interesting is that writing a small "scene" and then hearing dialogue being spoken by a variety of voices often prompted the writing of further lines of dialogue in response to perceived emotion contained in voices in the generated output. Plus it was just fun. :)

The version of the tool on that page is based on Larynx TTS which has continued development more recently as Piper TTS: https://github.com/rhasspy/piper

I'm yet to publish my port which uses Piper TTS though: https://gitlab.com/RancidBacon/larynx-dialogue/-/tree/featur...

Though I did upload some sample output (including some "radio announcer" samples in response to a HN comment :) ): https://rancidbacon.gitlab.io/piper-tts-demos/

Obviously there's variations in voice quality, and ability to control expression is currently limited but beats hearing my own voice. :D

Mozilla's browser tts is kind of not bad, just parse and buffer one sentence at a time and it does all right.

For the backend, I've experimented with piper, which has a lot of voices and accents, though it's tricky to buffer and sync long texts.

https://github.com/rhasspy/piper

After some brief research it seems the issue you're seeing may be a known bug in at least some versions/release of espeak-ng.

Here's some potentially related links if you'd like to dig deeper:

* "questions about mandarin data packet #1044": https://github.com/espeak-ng/espeak-ng/issues/1044

* "ESpeak NJ-1.51’s Mandarin pronunciation is corrupted #12952": https://github.com/nvaccess/nvda/issues/12952

* "The pronunciation of Mandarin Chinese using ESpeak NJ in NVDA is not normal #1028": https://github.com/espeak-ng/espeak-ng/issues/1028

* "When espeak-ng translates Chinese (cmn), IPA tone symbols are not output correctly #305": https://github.com/rhasspy/piper/issues/305

* "Please default ESpeak NG's voice role to 'Chinese (Mandarin, latin as Pinyin)' for Chinese to fix #12952 #13572": https://github.com/nvaccess/nvda/issues/13572

* "Cmn voice not correctly translated #1370": https://github.com/espeak-ng/espeak-ng/issues/1370

If you're not already aware, the primary developer of Mimic 3 (and its non-Mimic predecessor Larynx) continued TTS-related development with Larynx and the renamed project Piper: https://github.com/rhasspy/piper

Last year Piper development was supported by Nabu Casa for their "Year of Voice" project for Home Assistant and it sounds like Mike Hansen is going to continue on it with their support this year.

Coqui-ai was a commercial continuation of Mozilla TTS and STT (https://github.com/mozilla/TTS).

At the time (2018-ish), it was really impressive for on-device voice synthesis (with a quality approaching the Google and Azure cloud-based voice synthesis options) and open source, so a lot of people in the FOSS community were hoping it could be used for a privacy-respecting home assistant, Linux speech synthesis that doesn't suck, etc.

After Mozilla abandoned the project, Coqui continued development and had some really impressive one-shot voice cloning, but pivoted to marketing speech synthesis for game developers. They were probably having trouble monetizing it, and it doesn't surprise me that they shut down.

An equivalent project that's still in active development and doing really well is Piper TTS (https://github.com/rhasspy/piper).

There isn't an ElevenLabs app like that, but I think that's the most expedient method, by far.

(details and warning: in-depth, opinionated take, written almost for my own benefit, I've done a lot of work near here recently but haven't had to organize my thoughts until now)

Why? Local inference is hard. You need two things: the clips to voice model (which we have here, but bleeding edge), and text + voice -> speech model.

Text to voice to speech, locally, has excellent prior art for me, in the form of a Raspberry Pi-based ONNX inference library called [Piper](https://github.com/rhasspy/piper). I should just be able to copy that, about an afternoon of work!

Except...when these models are trained, they encode plaintext to model input using a library called eSpeak. eSpeak is basically f(plaintext) => ints representing phonemes. eSpeak is a C library and written in a style I haven't seen in a while and depends on other C libraries. So I end up needing to port like 20K lines of C to Dart...or I could use WASM, but over the last year, I lost the ability to be able to reason through how to get WASM running in Dart, both native and web.

It's a really annoying technical problem: the speech models all use this eSpeak C library to turn plaintext => model input (tokenized phonemes).

Re: ElevenLabs

I had looked into the API months ago and vaguely remembered it was _very_ complete.

I spent the last hour or two playing with it, and reconfirmed that. They have enough API surface that you could build an API that took voice recordings, created a voice, and then did POSTs / socket connection to get audio data from that voice at will.

Only issue is pricing IMHO, $0.18 for 1000 characters. :/ But this is something I feel very comfortable saying wouldn't be _that_ much work to build and open source with a "bring your own API key" type thing. I had forgotten about Eleven Labs till your post, which made me realize there was an actually meaningful and quite moving use case for it.

piper (https://github.com/rhasspy/piper)

Funny, I just picked up a device for use with https://heywillow.io for similar reasons

Fair points but with all due respect completely misses the point and context. My comment was a reply to a new user interested in esphome on a post about esphome.

You're talking about CircuitPython, 35KB web replies, PSRAM, UF2 bootloader, etc. These are comparatively very advanced topics and you didn't mention esphome once.

The comfort and familiarity of Amazon for what is already a new, intimidating, and challenging subject is of immeasurable value for a novice. They can click those links, fill a cart, and have stuff show up tomorrow with all of the usual ease, friendliness, and reliability of Amazon. If they get frustrated or it doesn't work out they can shove it in the box and get a full refund Amazon-style.

You're suggesting wandering all over the internet, ordering stuff from China, multiple vendors, etc while describing a bunch of things that frankly just won't matter to them. I say this as someone who has been an esphome and home assistant user since day one. The approach I described has never failed or remotely bothered me and over the past ~decade I've seen it suggested to new users successfully time and time again.

In terms of PSRAM to my knowledge the only thing it is utilized for in the esphome ecosystem is higher resolution displays and more advanced voice assistant scenarios that almost always require -S3 anyway and are a very advanced, challenging use cases. I'm very familiar with displays, voice, the S3, and PSRAM but more on that in a second...

> live with one less LX7 core and no Bluetooth

I'm the founder of Willow[0] and when comparing Willow to esphome the most frequent request we get is supporting bluetooth functionality i.e. esphome bluetooth proxy[1]. This is an extremely popular use case in the esphome/home assistant community. Not having bluetooth while losing a core and paying more is a bigger issue than pin spacing.

It's also a pretty obscure board and while not a big deal to you and I if you look around at docs, guides, etc, etc you'll see the cheap-o boards from Amazon are by far the most popular and common (unsurprisingly). Another plus for a new user.

Speaking of Willow (and back to PSRAM again) even the voice assistant satellite functionality of Home Assistant doesn't fundamentally require it - the most popular device doesn't have it either[2].

Very valuable comment with a lot of interesting information, just doesn't apply to context.

[0] - https://heywillow.io/

[1] - https://esphome.io/components/bluetooth_proxy.html

[2] - https://www.home-assistant.io/voice_control/thirteen-usd-voi...

> - People might criticize my messy/bad/unfinished code

As someone who has created and maintained open source projects (most recently Willow[0]) for two decades I get a kick out of this.

Of course when interacting with users and feedback I keep it polite but in my head I'm thinking "You like to talk. I actually DID this. Shut up or submit a PR".

Surprise surprise they almost never do.

[0] - https://heywillow.io/

Also check out Willow- https://heywillow.io

It doesn’t synthesize voice back (yet) but open source and runs all offline on ESP32-based hardware and works with HomeAssistant!

Look into https://heywillow.io/ - still early in the project but they are getting good results.

Very nice!

Would you be interesting in integrating with my project Willow[0]?

Willow supports Home Assistant, OpenHAB, and generic REST+MQTT endpoints today. With Home Assistant and OpenHAB we benefit from their specific API support for providing speech to text output and processing through things like the HA Assist Pipelines[1].

From our standpoint we handle wake word, VAD+AEC+BSS, STT, TTS, user feedback, etc. All we really do is send the speech transcript to the Willow command endpoint (like HA) and speak+display the execution result. Other than all of the wild speech stuff and our obsession with speed and accuracy Willow is really quite "dumb" - think of it as a voice terminal.

OpenHAB has something similar but it's significantly more limited.

[0] - https://heywillow.io

[1] - https://developers.home-assistant.io/docs/voice/pipelines/

I'm the founder of Willow[0] (we use ctranslate2 as well) and I will be looking at this as soon tomorrow as these models are released. HF claims they're drop-in compatible but we won't know for sure until someone looks at it.

[0] - https://heywillow.io/

Shameless self-plug but with my project Willow[0] we have a management server implementation to deal with multiple devices, etc. We have a new feature called "Willow One Wake" that takes the incoming audio amplitude when wake word is detected and uses our Willow Application Server (python) to only activate wake on the device closest to the person speaking. Old and tired compared to the commercial stuff but a first in the open source space.

The asyncio improvements in Python 3.12 especially (plus perf generally) have been instrumental in enabling real world use of this. With Python 3.12 asyncio, uvloop, and FastAPI it works remarkably well[1]. As the demo video shows not only does it not delay responsiveness, it has granularity down to inches.

[0] - https://heywillow.io/

[1] - https://youtu.be/qlhSEeWJ4gs