etl4s 2.x: Reimagining the Data Pipeline with Free Arrows

In etl4s 1.x, if you stitched two blocks with ~>, you got back a third. You could run it, and see its config requirement if wrapped in a Reader monad… but that was it. You couldn’t draw it, test its shape, or trace the blast radius of a code change without running it.

etl4s 2.x keeps the exact same API: but when you stitch you don’t get an opaque function, you get a Free Arrow: an AST describing the pipeline itself… without prescribing a compute engine like Spark/Flink, or having you bend the knee to a “how you ought to do things” framework like Beam.

Val names bound to nodes, unerased types, and FQCNs are captured at compile time.

val p = a ~> b ~> c

That’s the AST of a ~> b ~> c. An intuitive chaining operator like the F# |>… except our pipeline is no longer just something you execute. It’s a value you can take apart.

etl4s’ Sleight of Hand

Discreetly (and deliberately), etl4s is a library with one core abstraction, Node[-In, +Out]… and you stitch these Nodes into your pipeline graphs with intuitive symbolic operators in infix position. So far, simple.

The first little bit of subterfuge is that those graph-building operators behave exactly the same on Nodes irrespective of whether or not they are wrapped in a Reader monad. ~> doesn’t care: plain Node or Reader[Env, Node], you stitch them together the same way.

The second little bit of subterfuge is that etl4s looks like an effect system, or at the very least has an API that gives the secret handshake to “effectitians” (unsafeRun, laziness). It doesn’t have an effect-row, or actually force nodes to be pure though.

Despite looking like an effect system to those who can “Speak friend, and enter”… it is literally being used by teams of 20+ engineers who would not identify as functional programmers, know about effect systems, or necessarily even know what Scala’s for comprehensions desugar to. And this in my opinion is a fairly interesting toe dipped into what could be a deeper research programme to “see how to bring the benefits of effect systems to non FP folks or PL geeks”.

Free arrows

Late in 2025, cross building etl4s to Native and JS was becoming a situation unto itself, and not one I was proud of. I set about making the library effect polymorphic, without necessarily knowing “Free” or “Arrow” would be at the core of the deep internal refactor.

To make a long story short, the trick behind 2.x is representing composition as data rather than immediately composing functions. There’s plenty of literature in the neighbourhood, and I eventually followed the spool of yarn back to Hughes’s Generalising Monads to Arrows, then Lindley, Wadler and Yallop’s wonderfully titled Idioms are oblivious, arrows are meticulous, monads are promiscuous, and sideways to Capriotti and Kaposi’s Free Applicative Functors (free applicatives rather than arrows, but the same “keep the structure, interpret later” idea).

In reality though, I first came into contact with “Free” as a concept when working as a programmer for Bell Canada around March-April 2022 and window-shopping at the (well-documented) Cats typeclasses. I quite simply do not have this type of Master’s+ background in mathematics.

For our purposes though, “Arrow” because we’re composing typed A => B computations. “Free” because we keep that composition as a tree instead of deciding how to execute it…

sealed trait Node[A, B]

case class Step[A, B](f: A => B) extends Node[A, B]

case class AndThen[A, B, C](
  left: Node[A, B],
  right: Node[B, C]
) extends Node[A, C]

Now ~> doesn’t have to run or even combine the underlying functions. It just builds another node.

a ~> b ~> c

Funnily enough, this was actually the very first suggestion I ever got when talking to strangers about etl4s on the programming interwebs… and I didn’t dismiss it in the negative sense… but my knee jerk reaction (a bit shamefacedly) to something I hadn’t used (Arrow) was: “This seems like some Bartosz Milewski self-indulgent stuff”. Don’t @ me, I own and have enjoyed his book, but it is good for programmers to write lines of production code.

Why profunctor?

Node[-A, +B] is literally the one core abstraction in etl4s… contravariant in, covariant out. That’s exactly the shape of a profunctor: it takes things in on one side and hands things out on the other. Scala lets you put that right into the type with declaration-site variance, which really is a thing of beauty (not the point of this blog though).

trait Animal
case class Cat(name: String) extends Animal

val getCat: Node[Unit, Cat] = ???
val processAnimal: Node[Animal, Animal] = ???

val p: Node[Unit, Animal] = getCat ~> processAnimal

Every Cat is an Animal, so a node that produces cats can feed straight into one that accepts any animal. The compiler works it out.

Tangentially related (and not the main point of this blog) but I came across this video of Simon Peyton Jones talking about parametric polymorphism and subtyping and he asks a pretty interesting question “Is the baggage of subtyping AND ad-hoc polymorphism worth it? (since they do overlap a lot)”

In my opinion, this facet of etl4s with the Node variance, combined with tricks 2/3 below answers a resounding “yes” to Peyton Jones’ question. I think retroactive polymorphism + subtyping are not just worth having, but sumptuous together.

The tricks of etl4s

What makes etl4s feel like etl4s comes down to three little tricks, working together like a phalanx:

None is particularly earth-shattering on its own, but together they account for much of what makes the library pleasant to use.

Trick 1/3: Stealthy Reader Monads

This is actually one of my favourite bits, and it predates 2.x. In early 2024, I spent a 72-hour long weekend coding up the part of etl4s that lets environment types propagate automatically. This was back when LLMs couldn’t write a Scala typeclass to save their lives, I say with some pride. I had Robert Palmer’s I Didn’t Mean to Turn You On playing on an endless loop and was demolishing bags of sour candy.

The problem is fairly mundane. Most pipeline steps need something from the outside world: a database handle, an S3 client, some configuration. Passing these around explicitly clutters call sites with things that have nothing to do with the actual flow of data.

etl4s uses Reader monads to deal with this (the details aren’t particularly important):

val extract: Reader[Db, Node[Unit, Row]] =
  Node[Unit, Row].requires[Db] { db => _ => db.fetch() }

val load = Node[Row, Unit].requires[S3] { s3 => row =>
  s3.put(row)
}

The .requires[Db] shorthand gives us a Reader[Db, Node[Unit, Row]]. I originally added it to hide the Reader machinery from newcomers, but quickly became rather fond of not having to hand-roll monad stacks or clutter my own type signatures.

In this example, extract needs a Db and load needs an S3. Fairly unremarkable.

val p: Reader[Db & S3, Node[Unit, Unit]] = extract ~> load

You can keep composing steps, each with its own requirements, and the environment type grows to accommodate them. More importantly, the same ~> operator works whether a step needs an environment or not. From the programmer’s perspective, it’s all just pipeline composition.

There’s no compiler plugin involved, either. Just a small typeclass and some carefully arranged implicit priorities:

trait ReaderCompat[A, B, R] {
  def toA(r: R): A
  def toB(r: R): B
}

The typeclass accounts for four cases. Both steps might need the same environment, one might already have everything the other needs (in either direction), or their requirements might need to be combined:

ReaderCompat[Db, Db, Db]
ReaderCompat[Full, Db, Full]
ReaderCompat[Db, Full, Full]
ReaderCompat[Db, S3, Db & S3]

Composition then projects the shared environment into whatever each step expects.

Reader { env =>
  fa.run(compat.toA(env)) ~> fb.run(compat.toB(env))
}

In a very “Clean Code”/ SOLID way, these env-propagating Reader monads make the config requirements follow the pipeline instead of the other way around.

You can build a fairly large pipeline out of small, independently defined steps, and literally be told at the typesystem level what to give it at the very end.

That’s really the whole trick. The programmer writes extract ~> load, and the type system takes care of the bookkeeping.

Trick 2/3: Grabbing val names at compile time

So an AST of anonymous functions would be a diagram full of <function1>s (not very handy). Under the hood etl4s grabs some information at compile time (with macros) before it is gone in bytecode.

def deriveImpl(using Quotes): Expr[Name] = {
  import quotes.reflect.*
  val sym = find(Symbol.spliceOwner)
  '{ Name(${ Expr(sym.name) }, ${ Expr(sym.fullName) }) }
}

Symbol.spliceOwner is wherever the macro got expanded, find walks up from there to the val, and '{ ... } is the code we hand back to the compiler, with ${ ... } dropping the names in as plain string literals.

Trick 3/3: (Familiar) Two-Ended Pipes

etl4s espouses the idea that it is useful to banish the assignment = operator at the key wiring stage of a dataflow program.

for {
  e <- extract
  t <- transform(e)
  _ <- load(t)
} yield ()

for {
  startDate = ???
  e         <- extract
  endDate   = ???
  t         <- transform(e, startDate)
  _         <- load(t, endDate)
} yield ()

Now your eyes go: startDate > e > endDate > t > (back to startDate) > load > (back to endDate)…

Push that config into the Reader’s type instead, and the wiring reads one way again:

extract ~> transform ~> load

Every node is ONE input, ONE output: a little two-ended pipe, not a multi-sided puzzle piece. You can still fan in and branch, but once stitched it’s a single node again:

(e1 & e2) ~> t >> save.If(_ > 0)(enrich ~> dbLoad)
                      .Else(process ~> purgatoryLoad)

We can take the critique of for comprehension syntax one step further in the sense that it is a shame that such a procedural, bamboozling to beginners notation has become the de-facto standard for “programs as values”.

Doesn’t Beam pipeline let you introspect pipeline graphs?

Sure, and so does Spark technically, but let’s take a step back… because they target vastly different problems.

I first got my hands on the excellent book Streaming Systems in 2021. The authors are passionate, right out of the gate they start explaining how they made their beautiful diagrams showing watermark nuances:

This is the type of thing that warms the heart of any geek. They have this quirky and very fun game show style of passing on the mic to the next co-author:

Which is a little funny at baseline, but funnier given we were already generating heartbeats for out-of-order streams back in 2004, and we were calling them low watermarks by 2008. Akidau and co. acknowledge this prior art in their Dataflow paper, but it wouldn’t have hurt to see a bit more of this history in the book.

They do a great job at presenting the Dataflow model, and Beam is the practical implementation of that. And yes, a Beam pipeline is a graph. You can walk it, the runners optimize it, the Dataflow UI draws it.

But it is a graph of Beam’s world. PCollections, PTransforms, ParDos, coders, windowing strategies. To get the graph you first have to bend the knee to their (intellectually impressive, and fairly useful) World View. But you still bend the knee.

You rewrite your logic in the SDK’s vocabulary, then hope the runner you picked speaks enough of that vocabulary.

This brings us to the most woeful part of the “I’ll use big interpreter-style frameworks with World views to reify my dataflows” approach … the capability matrix:

Credit where it’s due… this is an honest page (bring back the dots). But look at all those ~. “Portable” turns out to mean “portable, partially, depending”.

etl4s goes after a much smaller and more boring (but in my opinion more widespread and corrosive) problem… messy un-reified pipelines that do not have typed, declarative endpoints.

With etl4s there is nothing to migrate to, no framework or World view to adopt: if a step wants to call Spark, it calls Spark. If it wants to hit Postgres with JDBC, it does that. The graph etl4s gives you is the graph of your code, with your val names on it, not the graph of the IR of some engine’s execution plan… made by a framework maker with a vested interest in getting their claws into your code.

So what do you actually get from etl4s?

val p = extract5 ~> (double & triple) ~> combine ~> saveToDb

p.unsafeRun()
p.compile[Try].unsafeRun()
p.compile[Future].unsafeRun()

Id, Try, and Future ship out of the box. Implement etl4s.Effect once and it runs on cats-effect IO, ZIO, Kyo… whatever. That was the whole reason I started tearing up the internals in the first place, and it fell out of the Free Arrow almost by accident.

Once the pipeline is a tree, “how do I run this” is just one more interpreter. And once you’ve got one interpreter, the next ones come cheap:

All of which, a bit humblingly, is more or less what that redditor said back in 2024. Give etl4s a spin. Ossu 🥋🙇, and happy hacking.