# Squidworks: Distributed Dataflow Control

The **Squidworks** project is a bundle of modular hardware and software that implements *Distributed Dataflow Machine Controllers*. These are a kind of controller whose constituent parts are located throughout a network, with no explicit central state machine. Since the coordination of networked computing resources is often opaque, squidworks proposes to implement virtual dataflow engines for modular code in each of these resources, to align code configuration and execution with that of the network itself.

![explainer](video/squidcut-sept2019-brief-wordless.mp4)

![works](video/2019-11-13_saturn-works.mp4)

Because our computing - and hardware - resources are modular, *adding and removing capability to a system should become easy.* It also means that *re-writing machine controllers,* to accept new inputs, or deliver new outputs (material, or feedback) should be simpler. Machine controllers are transparent: their operating principles are obvious to their users, and intelligent intervention (and modification!) of controllers should be possible. We can re-use algorithms and processes that we have developed across many machine instantiations in any particular case, meaning that we should be able to build unique systems with less overhead. Digital tools are pliable again, and under our control.

## How does it Work?

### Virtual Dataflow Environments

Each of the processors (big or small!) participating in the control of these machines runs a *virtual dataflow environment.* These are a kind of operating system that simulates network operation between modular pieces of code. Codes are self-contained programs that interface to other codes via data inputs and outputs: we can think of these as virtual data ports.

[Cuttlefish](https://gitlab.cba.mit.edu/squidworks/cuttlefish) is one such VDE, for the browser. It also serves visual representations of its own dataflow graphs, as well as dataflow across a network. This is the tool that allows us to see, interact with, build and edit distributed programs.

[Ponyo](https://gitlab.cba.mit.edu/squidworks/ponyo), the smallest fish (and queen of the sea), runs dataflow graphs on embedded microcontrollers. At the moment this is specific to the ATSAMD51J19, a 120MHz Arm M4F.

### Links, and the Common Serialization

To nest virtual dataflow environments inside of networks (which are 'real' dataflow environments!) Squidworks develops a common serialization for known datatypes (i.e, in network byte-space, what does it mean to send a 64-bit float from port:0 in context:a to port:2 in context:b - etc), as well as common descriptors for graph configurations (based on the same serialization scheme).

Squidworks networks aren't picky about physical connections: these can be (and often are) USB links, RS-485 UARTs, and WebSockets, and could be (but are not yet) Bluetooth, LoRA, Laserbeams, Wiggled-Rope, Deep-Sea Cable, Interstellar Radio Link, etc - physical links. Anything that can ferry bytes from one context to another will do.

## Circuits

At the end of the day, if we're still just talking about machine control, current moves through coils, voltages are read and delivered, etc. These machines aren't going to operate themselves (or, uh oh...) - so the squidworks project umbrella's real physical work on developing the hardware tools we can use to make them dance.

| **The Router** |
| --- |
| A message passing device, this thing hooks 6 of the ATSAMD51's SERCOM USARTS up to RS-485 Differential Driver and then hooks those up to RJ10 connectors (read: phone handset jacks). It runs [ponyo](https://gitlab.cba.mit.edu/squidworks/ponyo) and you can see the [schematic, board and documentation here](https://gitlab.cba.mit.edu/squidworks/routerboard-atsamd51). |
| ![router](images/2019-11-17_router.jpg) |

| **The Module Board** |
| --- |
| A do-what-you-will-with-it device, this thing breaks all of the ATSAMD51's pins out to fab-lab-friendly sized castellated pins, so that you can solder it to some-circuit-of-your-design. The thing is ready to run [ponyo](https://gitlab.cba.mit.edu/squidworks/ponyo), and invites you to, indeed, write some CPP and integrate some new devices onto the network that it will happily join over that RS-485 link. [Schematic, board and documentation here](https://gitlab.cba.mit.edu/squidworks/moduleboard-atsamd51). |
| ![module](images/2019-10-30_mw-module.jpg) |

| **The Steppers** |
| --- |
| A motor-turning device, this thing is one of the aforementioned module-boards, soldered to a heavy duty, motor-wrastling, no-amps-barred TMC262 stepper driver which *itself* slices and dices 24v of *power* with the help of four (4!) PN-Pair mosfets (that's two whole h-bridges baby) to drive (probably) NEMA17 stepper motors, to which these things will be attached when you get them. This also runs [ponyo](https://gitlab.cba.mit.edu/squidworks/ponyo) and you can see the [schematic, board and documentation here](https://gitlab.cba.mit.edu/squidworks/moduledaughter-stepper). |
| ![stepper](images/2019-11-18_stepper.jpg) |

| **Power Distribution Bits** |
| --- |
| These are tiny bus-bar type devices, that should make power routing a little bit easier. Included in the kit are (1) bypass capacitors for spare charge (these are actually important for the stepper motors to work properly), (2) TVS Diodes and Bleeders, (3) 5V Regulators (also necessary) and (4) routing blocks. These are all documented and explained in [this power distribution repo](https://gitlab.cba.mit.edu/squidworks/powerdistribution). |
| ![psu](images/2019-10-30_mw-psu.jpg) |

## What is the Status of the Squidworks Project?

The project is relatively nascent, but ongoing, and we are excited to continue its development. Nothing is set in stone, but most components are annealing well. While the project proposes radical re-usability of code- and hardware resources, designing software to be appropriately general takes some careful thought, and as a result the software isn't yet appropriate for individuals who do not know how to, or do not want to learn how to program: you will still find that accomplishing most tasks requires your developing some custom 'hunks' (our modular code blocks). Stay tuned, and try it out now if you're feeling adventurous and curious.

## How can I build a Squidworks Controller?

To start, you can download and run [cuttlefish](https://gitlab.cba.mit.edu/squidworks/cuttlefish), and then acquaint yourself with [ponyo](https://gitlab.cba.mit.edu/squidworks/ponyo). It's possible (though not tested) to build Ponyo for almost any embedded device (no part of 'ponyo' itself is tied to hardware directly), but if you're looking to start fast, we recommend [Adafruit's Feather M4](https://www.adafruit.com/product/3857), which is the board we developed it on. In fact, all of our circuits still think they are Feather M4s (we are still using their bootloader: thanks Adafruit!).