Building distributed ecosystems is all about traded simplicity. As the applications become smaller and more single-purposed, they become easier to develop and maintain. Unfortunately, the trade-off is a much more complex operating environment where there is no single direction of execution. Essentially, almost anything can send and receive data from anywhere else. Being able to conceptually understand how all the pieces fit together as well as how each individual piece can fulfill (for lack of a better term) a "distributed computing contract" will go a long way towards being able to build out and maintain these types of systems.
Over the course of this post, as well as in planned follow-ups, we will explore how to build powerful cloud-native applications using some basic, distributed building blocks. Some of the algorithms will work fine in serverless environments; some will be more adept in distributed, containerized environments. In all of the cases though, you should be able to extrapolate these ideas to solve all sorts of sticky situations when you delve into the world of loosely-coupled, concurrent systems.
Building the Foundation
To begin, we need to be able to do more than one thing at a time reliably, and preferably, do them simply.
In a nutshell, we want:
- The ability to route and respond to basic web services
- The ability to dispatch work into the background to be performed based on timing, as well as in response to events
Typically, these two primary functions are talked about in separate circles, and for good reason - they have different purposes. In a coarse sense, one is interactive (web services) and the other is batch (background processes). We want to bring these together in order to show off how some very powerful distributed concepts, can be used internally to provide better and more flexible applications, while still being easy to maintain.
A side-effect of these techniques will be constructing a local playground in which we can test out distributed patterns and cloud-native concepts without the need for any external infrastructure or connectivity. That will be demonstrated in detail later on.
To assist us, we will leverage Google's golang programming language and its concurrent primitives to add some fun, yet powerful capabilities that make an application more suited for distributed cloud environments. Additionally, these constructs will be useful in any environment where additional "situational awareness" will lead to a more predictable, flexible, and scaleable application architecture.
Application: A Concurrent Web Services Engine
Our first goal is to build a basic application that will act primarily as a web services router with the extra ability to generate a time event as a separate process.
Our single application is actually going to be two applications (web & worker) combined in a single binary. Both processes will be accessible to us through a services interface and both processes will be aware of each other.
Why Does Our Background Processor Need to Handle Web Services?
Typically, a web service application is designed to answer REST calls coming from some sort of front-end (e.g. web, voice, other web service) calculate an answer based on internal knowledge or the result of a database query - then send it back along the same channel from which the request originated. Preferably along with a 200 as an HTTP response code. But sometimes, the conditions in which the application lives or the realities of the tasks it needs to perform require some additional flexibility.
In the case of our future distributed applications, we want to be able to pull metrics from running systems so that we can better monitor the progress of data going through the application and modify its behavior based on changing business needs. Sure, we could spin up workers with different capabilities, but we're looking for a more dynamic approach in this instance by changing behavior on the fly. Plus, it lets us play with different distribution models without needing the corresponding infrastructure.
Initialize the Program
First, we need to set up the application. Using the init() function, we set up the basic stuff we need for execution. For this first application, we will hardcode a port for simplicity, but in future applications, we will use either an environment variable or be able to accept a container driven value to specify the appropriate port.
// Do all of this stuff first.
func init() {
// In this example, we will hard code the port. Later the environment
// will dictate.
port = 7718
// Set up the heartbeat ticker.
heartbeat = time.NewTicker(60 * time.Second)
// Setup the service router.
gin.SetMode(gin.ReleaseMode)
router = gin.New()
// These are the services we will be listening for.
router.GET("/beats", GetHeartbeatCount)
router.GET("/ping", PingTheAPI)
} // func
Additionally, here we instantiate a router to handle the web services requests - in this case Gin, and specify two routes: one for returning the number of heartbeats the application has performed since it was started, and a "ping" service that returns a simple "PONG" whenever the ping service is called. The former service allows us to get metrics from the background process, whereas the latter is a quick way to make sure the application is functioning enough to answer simple requests.
The functions that are called by the router are detailed below. Essentially, they both return a 200 and either a variable counter in the case of GetHeartbeatCount() or a static string for PingTheAPI(). All we aim to show here, is that we have access to both of the primary processes of the application: the foreground and the background.
// GetHeartbeatCount sends the number of times the heartbeat ticker has
// fired since the program started.
func GetHeartbeatCount(c *gin.Context) {
content := gin.H{"payload": beats}
c.JSON(http.StatusOK, content)
}
// PingTheAPI lets the caller know we are alive.
func PingTheAPI(c *gin.Context) {
content := gin.H{"payload": "PONG"}
c.JSON(http.StatusOK, content)
}
Now, let's talk about what goes into the background piece.
Why Does Our Web Service Need a Background Processor?
In a distributed environment, there may be some mitigating circumstances where it would be nice for an application to be able to make decisions or exercise some corrective action based on changes in the surrounding environment. As an example, upstream or downstream connection points not being available, traffic back-ups that require the application to scale up/down to smooth out transaction flow, or the ability to change its own behavior based on incoming data. Doing this reactively through a web service or even proactively via an outside orchestration layer can be workable solutions, but what if we want our applications to be a bit more self-aware or autonomous?
Golang is obviously not the first, nor is it the only language that handles for concurrent programming. However, it is arguably one of the easiest and more robust tools with which to employ it.
Some reasons why we may want to add web service capability to a process that is traditionally, for lack of a better term, a "batch" routine:
- We want to pull status or operating metrics from the process while it is running without impacting the application's primary function
- We want to change the behavior, rules, or configuration of a running process
- We want to override operations (e.g. force scaling) on the application irrespective of current conditions
How We are Going to Achieve It
An application heartbeat is good for the same reason a heartbeat in any living thing is good - it shows you're alive. For our application, a heartbeat can be used to keep a stream connection alive, send data to a downstream connection on a timed basis, or simply just be the pulse to show that everything is still okay. If the pulse stops, it may be a good indication that our little application needs to be garbage collected by its higher power (e.g. its container).
Setting Up the Heartbeat Ticker
There are many ways we can set up an application heartbeat. The easiest way is just to have an external source periodically ask if our app is OK via a "health-check" service. If our app responds appropriately, then the service that is doing our overwatch keeps the little status moniker green at least until the next health-check. Unfortunately, this is a very reactive approach. We want a more proactive approach because in a distributed environment, we may want to send out signals when stability starts to wobble in our application's situation so that impacted systems can alter their behavior accordingly. It's all about being a good nodizen or a nodal citizen (Whichever pun works for you. Essentially both represent a distributed node that acts in the best interest of the overall environment by doing its part to maintain its own stability, and those nodes upstream of it). Luckily, Go has a several built-in types that allow us to take the proactive approach more easily. One of these is the ticker.
A ticker in Go is simply a timer that runs forever, but for every specified unit-of-time (from microseconds to years), it sends a signal across a channel internal to your application. If you are watching this channel, you can execute code every time you see that signal. If you're not watching for it, the ticker will happily throw that signal into the ether. It's very pub/sub that way.
To set up the ticker, you simply define it, create it, and give it an appropriate interval. In this example, we create a ticker called heartbeat and tell it to fire every 60 seconds.
heartbeat *time.Ticker
heartbeat = time.NewTicker(60 * time.Second)
Watching for the Heartbeat
So we have a heartbeat happening somewhere in the depths of our application, now we need to do something to surface that signal so that we can act upon it. Happily, this is a simple process as well. Behold!
<-heartbeat.C
That's it! All this statement says is "watch the heartbeat channel." Something will only come across this channel when the ticker fires (in this case, once per minute), and we will need to constantly watch it in order to catch that action - so while one line of code is simple, we will need a little bit more to make watching this channel workable.
// Dispatch a process into the background.
go func() {
// Now run it forever.
for {
// Wait for stuff to happen.
select {
// When the Heartbeat ticker is fired, execute this.
case <-heartbeat.C:
beats++
fmt.Printf("bump,Bump...\n")
} // select
} // for
}() // go func
Now we have what we need in order to dispatch a worker to the background, but still let it have direct access to the foreground and vice versa.
Outputting the Proof
The application is coded, so let's test it! Find a command line, then:
- Build and install the binary with a
go install - Run the application:
gosp1
We should see something like this: ready on port 7718. If you changed the port number in the source, use that here. If we continue watching the application from the run location, eventually we should see a heartbeat pop up every minute.
ready on port 7718
bump,Bump... @ 2018-08-20 14:38:03.257404246 +0000 UTC
bump,Bump... @ 2018-08-20 14:39:03.257577644 +0000 UTC
bump,Bump... @ 2018-08-20 14:40:03.257841167 +0000 UTC
bump,Bump... @ 2018-08-20 14:41:03.258481854 +0000 UTC
bump,Bump... @ 2018-08-20 14:42:03.258159098 +0000 UTC
bump,Bump... @ 2018-08-20 14:43:03.258863155 +0000 UTC
Seeing this, we know the background service is running as it should be. Every minute (give or take a millisecond), the heartbeat ticker fires and our heartbeat code is executed. The question now becomes, is our little application also listening for services? Let's check via curl.
curl http://localhost:7718/ping yields {"payload":"PONG"}
Excellent! Now, are we getting updates from the heartbeat process? Let's check by do the same thing. Using curl, hit the beats web service.
curl http://localhost:7718/beats yields {"payload":6}
Yes, we are indeed getting data derived from the actions of the background process. This is going to be important later when things get more complicated. Full code can be found on GitHub.
Conclusion
This is the baseline pattern we will be using to build out some more complicated stuff in the next few posts. Granted, this example shows a very simple pattern doing very simple things - but we need to establish this baseline in order to know that the rest of what we want to build will have a proven foundation.
As we move down the path, we will also cover some of the "gotchas" that can crop up when dealing with concurrency in general, and in Go in particular. At this point, we are in good shape for building on this foundation moving forward.
Up next, we will extend this pattern to show how a model distributed application ecosystem can be contained in a single application. Later, we will extend that exercise into even more complex examples involving circuit beakers, receipt processing, and digital twins.
