Folie is growing up... slowly

Folie, the “Forth Live Explorer” is starting to shape up. Here’s a summary of what it’s about:

• a front end for Mecrisp Forth, i.e. a terminal emulator like picocom or Tera Term
• easily find (ctrl-R) and re-enter previous commands (up-/down-arrow)
• send source files to Mecrisp Forth as if they had been literally typed in (!send)
• expand include directives in the source code with the file(s) they refer to

The above will work with any serial connection, both RS232 and USB, as well as with a network connection via ser2net. For additional functionality, a BUB III, a modified BUB, a SerPlus, or a Telnet-enabled network connection via ser2net or an esp-link is required.

These all offer a way to control two additional signals: DTR + RTS, and with them, Folie can:

• reset an attached µC board whenever you press Ctrl-C (this only requires DTR)
• upload code to the STM32 µC using its built-in ROM-based boot loader

That last one is very handy if you have a shiny new STM32 board and need to get Mecrisp Forth onto it (or anything else for that matter). And while Mecrisp Forth is very robust, it is possible to add code in flash which will fail to start up after a reset, so the ability to upload is also useful to get a “bricked” board back into working order.

Folie is built as a stand-alone executable and its binaries should work out of the box on a wide range of Windows, MacOS, and Linux platforms - see the GitHub Releases page. For building from source you need to install the Go compiler, as described here. After that it’s as simple as:

go get -u github.com/jeelabs/folie

Building (i.e. cross-compiling) a version for the Raspberry Pi is a matter of typing:

GOOS=linux GOARCH=arm go get -u github.com/jeelabs/folie

One way to describe Go, is that it does what C & C++ never achieved: a fast, type-safe, statically compiled language with built-in concurrency primitives and an efficient garbage collector. Go was chosen for Folie because of its 100% self-contained executables, its multi-plaform build convenience, and its modest memory requirements - a great match for low-end Linux boards.

Folie version 2.11 was released a few days ago - it fixes some bugs (including a serious one introduced in 2.10) and greatly improves the upload process. Some more conveniences have been added, such as filename-completion with the !s (send) and !u (upload) commands.

All the special “!” commands will be listed when you enter !h’ or !help:

Special commands, these can also be abbreviated as "!r", etc:
  !reset          reset the board, same as ctrl-c
  !send <file>    send text file to the serial port, expand "include" lines
  !upload         show the list of built-in firmware images
  !upload <n>     upload built-in image <n> using STM32 boot protocol
  !upload <file>  upload specified firmware image (bin or hex format)
  !upload <url>   fetch firmware image from given URL, then upload it
Utility commands:
  !cd <dir>       change directory (or list current one if not specified)
  !ls <dir>       list contents of the specified (or current) directory
  !help           this message
To quit, hit ctrl-d. For command history, use up-/down-arrow.

Use the upload commands when you need to set up a new board or re-flash it at a later date. Uploads work with both hex and binary files (Folie will auto-detect the format), and can come from three different sources:

• upload from a file on your disk: !u <path/to/file>
• upload from a URL, i.e. fetching it over HTTP or HTTPS: !u <url>
• upload from a built-in firmware image: !u <n>

That last variant can be particularly convenient, since you don’t need to locate or download anything else once you have Folie. Here is the list of built-in firmware images, as of May 2017:

1: F103-BMP         50096b  crc:F87A
2: F103-Blink         724b  crc:4967
3: F103-Mecrisp     20500b  crc:A4D0
4: F103-SerPlus      7052b  crc:7DD0
5: L052-Blink         568b  crc:311E
6: L052-Mecrisp     20500b  crc:00AC

In Folie 2.11, the Mecrisp Forth images are now version 2.3.6 - one for F103 boards, the other for the JeeNode Zero. They present a serial console on USART1 (GPIO pins PA9 and PA10).

Rough edges

Folie has definitely reached a usable state, at least on MacOS and Linux. On Windows, we’re not quite there yet, it seems - though the go-serial and readline packages it uses have been steadily evolving and progressing. While Folie 2.11 should again improve the situation on Windows, you can definitely help by trying this out and reporting any quirks/issues on GitHub.

The other tricky aspect of Folie, is that it tries to out-guess Mecrisp Forth’s command line and prompt, and it doesn’t always get it right. If you’ve used Mecrisp Forth, you’ll have seen that it has a somewhat unusual prompt: “ok.” gets printed at the end of the line it just processed. This is in fact very common in Forth systems… it’s just not the way most applications work today.

This is how Folie interacts with Mecrisp’s command line and prompt:

• the host (i.e. Folie) sends a command, terminated with a carriage-return (CR)
• Mecrisp echoes the characters, but echoes the CR as a space
• the Forth code is evaluated, and possibly generates some output
• when done, Mecrisp prints a space, “ok.”, and a line-feed (no CR)
• rinse and repeat…

The tricky part comes from the fact that Folie uses the “readline” package, which supports line editing, history recall, and more. Which is why Folie sends the entire line at once, appending CR as last character. So Folie is not a char-by-char terminal emulator, but a line-based one.

Because of this, local editing and character display has already happened and the echo produced by Mecrisp needs to be supressed. This is relatively simple:

• after sending the line + CR, Folie waits for the line + space to come back
• everything else gets sent to the screen, and must be other output
• the line + space is read and discarded
• then Folie reads all text until space + “ok.” + LF is seen
• this too is discarded, unless there is other output as well, then it all gets printed

One complication is that not all commands end with an “ok.” prompt from Mecrisp:

• the reset command sends a greeting, but no terminating “ok.”
• so do eraseflash and a few others, because they end with a call to reset
• commands can generate an error message - this is treated as any other output
• some commands take a long time to complete, or might even run forever

Folie uses a 1-second timeout to deal with this last case (in the sense of not waiting forever and allowing new input even in the absense of the “ok.” prompt). The result of all this works quite well, but there are still a few edge cases which cause the screen output to mis-behave.

One very important benefit of all this, is that Folie “knows” when Mecrisp is waiting for input, and refrains from sending text faster than it can be processed. This is why Folie’s include file expansion works really well - a complete source file can be sent without overflow errors, even when some of the lines take a bit more time to process. With more naïve solutions using plain terminal emulators, you need to throttle the sends to deal with the worst case slowdowns. Since Mecrisp needs more time to look up words as its dictionary grows, these delays can add up. With Folie, the rate matches Mecrisp’s needs regardless of µC speed and processing overhead.