Distributed Services with Go

Book PDF


We will design, develop, and deploy a distributed service to explore what Go can do. We’ll develop and deploy the service in layers: from the bare essentials of storage handling, to the networking of a client and server, to distributing server instances, deployment, and testing. I divided this book into four parts that parallel those layers.

Chapter 1: Test Your API


Chapter 3: ???

Each segment comprises a store file and an index file. The segment’s store file is where we store the record data; we continually append records to this file. The segment’s index file is where we index each record in the store file.

Index files are small enough that we can memory-map them and make operations on the file as fast as operating on in-memory data.

Memory-mapping isn’t necessarily size-constrained, so I’m not sure I follow. Data is still making it to disk eventually, so isn’t this simply trading explicit IO latency for implicit/hidden IO latency? I’m not seeing the benefit of using mmap here yet.


  • Record: the data stored in our log.
  • Store: the file we store records in.
  • Index: the file we store index entries in.
  • Segment: the abstraction that ties a store and an index together
  • Log: the abstraction that ties all the segments together.

The reason we resize them now is that, once they’re memory-mapped, we can’t resize them, so it’s now or never.

Why does the index store offsets? Isn’t an entry’s relative position in the index sufficient to determine its offset?

Chapter 4: ???

Internal packages are magical packages in Go that can only be imported by nearby code. For example, you can import code in /a/b/c/internal/d/e/f by code rooted by /a/b/c, but not code rooted by /a/b/g.

gRPC optionally supports status codes (not related to HTTP status codes