Availability Implementation In Go Service with Elastic APM

Tech Stuff Nov 24, 2021

Before going on and implementing our availability function in Go, let me say something about why we need to have it in the first place.

In a distributed and complex infrastructure, we basically have more than one service or application that runs independently from each other. It's at that level that we need to design and build a system where we can see the availability and or visibility of a system to make sure that when it's run in production, operators responsible for it can detect undesirable behaviors (e.g. service downtime, errors, and slow responses, to name a few) and have actionable information to pin down the root cause in an effective manner (e.g. detailed event logs and application traces).

Technically speaking, when we say availability, we only mention traces or flows of functionalities triggered when that specific service is on duty (I might cover logging in a different blog post). We need to make sure that every functional action traces back so that we may check for availability later on. Tracing should also include any logic interacting with external services (e.g., database interactions and caching service interactions).

We're going to use Elastic APM to monitor our Go API service. Elastic APM has agents for us to send data from services of any kind of language including Go, Java, Ruby, and etc.

Let's get started

Before starting to implement anything I'm using This Repository for this blog. You'll also find a docker-compose for the elastic stack containing elasticsearch, kibana, and an APM server. Without further ado, let's hop on to the fun part!

We should initially have a simple web service using a gorilla/mux router that has middleware registered to it. Doing that will report any request as a transaction to the APM server.

Adding middleware to our gorilla/mux router using: (Source)

// middleware using "go.elastic.co/apm/module/apmgorilla"
ssoRouter.Use(apmgorilla.Middleware())

Before running the web service try to export those values as follows:

$ # run the elastic stack using ...
$ # docker-compose file on different session using
$ docker-compose up
$ export ELASTIC_APM_LOG_FILE=stderr
$ export ELASTIC_APM_LOG_LEVEL=debug
$ export ELASTIC_APM_SERVICE_NAME=sso-service
$ # And then run the service using 
$ go run .

Let us now check what we get in our APM server when we try to query this endpoint: (i.e. endpoint is used to fetch JWKS keys).

# Endpoint
[GET] http://localhost:8181/api/v1/sso/jwks

Once we've queried our endpoint, we can go to http://localhost:5601/ and head to Observability > APM from the main menu of the Kibana dashboard to visualize our service (i.e. soo-service).

You can clearly see that the execution of our result from the endpoint took us 78 μs.

We've seen that we can visualize the performance of our service by just adding a single statement of passing our request through an APM-module middleware.

At some point, we also want to see the performance of interacting with external services (i.e., databases, caching, and so on). Being able to see the performance of the interaction with those external services is crucial.

Tracing communication with external services

As an example, our repository has an SQL interaction using go-sqlite3 but we should use a module 'go.elastic.co/apm/module/apmsql' to trace database SQL operations.

Here's how we've declared our driver to connect to our DB (we're using sqlite3 for simplicity here): (Source)

// using "go.elastic.co/apm/module/apmsql" &
// _ "go.elastic.co/apm/module/apmsql/sqlite3"
    
Db, err = apmsql.Open("sqlite3", "./sso.db")

We then explicitly pass our context initiated from our handler's request to the database function to track and trace the query made to our DB. Lucky for us we have an endpoint that utilizes a database interaction (i.e. http://localhost:8181/api/v1/sso/register):

# Endpoint
[POST] http://localhost:8181/api/v1/sso/register
JSON Data:
{
  "name": "TestName",
  "email": "[email protected]",
  "password": "password1234" 
}

That endpoint's handler will then pass the context of the *http.Request and pass it all the way to the database function.

# Passing our context
...
err = f.service.RegistrationService(r.Context(), user)

# Tracing our performance for our database query
func (r *repo) CreateUser(ctx context.Context, user *User) (err error) {
	tx, err := r.db.BeginTx(ctx, nil)
	if err != nil {
		return
	}
	stmt, err := tx.PrepareContext(ctx, "INSERT INTO users(name, email, password) values(?,?,?)")
	if err != nil {
		return
	}
	_, err = stmt.ExecContext(ctx, user.Name, user.Email, utils.Encrypt(user.Password))

	if err != nil {
		if sqliteErr, ok := err.(s3.Error); ok {
			if sqliteErr.Code == s3.ErrConstraint {
				err = errors.New("email already exist")
			}
		}
		return
	}
	return tx.Commit()
}

Finally, we'll get a performance report in our Kibana dashboard for our endpoint when it's been queried.