Terraform Mental Model

06.09.2021 09:23

Having learnt more about terraform, coupled with some hands-on at work, I have expanded my mental model of terraform.

Drawing it out on paper,

We can use count = [some number] to create multiple instances of each resource. If we ever need to identify each instance of a resource, we can make use of a contextual variable called count.index for the index of current instance.

In the diagram above, I have also illustrated a module. Terraform modules are used to organize and reuse logical units of terraform code. Each module can then can its own set of inputs (e.g. variables.tf), outputs (e.g. outputs.tf) and body (main.tf). These collectively describe a single resource or set of related resources (e.g., a docker container with a certain image and volume mount). The body (main.tf) of a module can also contain other modules. So we can see that modules provide us a layering and reuse mechanism for structuring a complex infrastructure.

Learning Terraform

02.08.2021 06:37

Recently I have been picking up Terraform on the job. I love the idea of Infrastructure as Code where all the resources are described in files that are versioned like regular code.

In this post, I am attempting to do a brain dump of what I have learned so far.

In short, I would describe Terraform as an IAC tool that lets you author files (.tf extension) that describes your infrastructure and helps you to manage (apply, destroy) the infrastructure state.

HDB Resale Pricing

11.07.2021 10:01

A few weeks ago, I started tinkering with our government’s HDB resale prices API . I had an idea to use recent transaction data to gauge if a listing on a property portal (e.g., property guru) is overpriced or fairly priced. Paralyzed by the sheer number of variables, I soon lost steam.

Resuming after a two week hiatus, I scoped the coding down to just fetching recent transactions which matched a listing’s address (specific block and street), flat type (4 room or 5 room) and floor level (low, mid and high).

Cl Json Path

06.06.2021 21:06

I extracted the JSON utility that I wrote previously into a standalone repository. Cleaned up the code and tests!

Check it out here!

Mortgage Calculator API

23.05.2021 20:47

I set myself a challenge to implement an API endpoint in Common Lisp, in hope of putting to use a few things that I have learnt recently,

defining structures
creating a new project using cl-project:make-project
adding and pulling in dependencies
serving REST endpoints using hunchentoot web server library
using rove testing framework to test drive implementation

And so, I settled on coding up a mortgage calculator which takes in principal (loan amount), tenure (in years) and annual interest rate, and returns a list of monthly payments, each comprising of interest, principal repayment, and outstanding principal so far.

Rest Helpers

11.05.2021 20:35

I extracted all the boilerplate to perform a REST get operation into rest-helpers:

(defun get-json (url)
    (let ((json))
        (ignore-errors   
            (multiple-value-bind 
                (data status) 
                (drakma:http-request url)
                (if (= 200 status) 
                    (let* ((str (flexi-streams:octets-to-string data)))
                    (setq json (json:decode-json-from-string str))))))
        (unless json (format t "~&Unable to fetch from ~A~%" url))    
        json))

Using both rest-helpers and json-helpers (from previous post), to get a list of astronauts in space requires much less code:

JSON Helper

03.05.2021 21:03

I wrote a helper get-json-value to extract values from json output produced by cl-json.


(defun property-p (property)
    (funcall #'consp property))

(defun get-property-value (property)
    (funcall #'cdr property))

(defun get-property-key (property)
    (funcall #'car property))

(defun json-object-p (json)
    (and (listp json) (every #'consp json)))

(defun json-array-p (json)
    (and (listp json) (every json-object-p json)))

(defun json-p (json)
    (or (json-object-p json) (json-array-p json)))

(defun get-json-value (keys json)
    (princ (list keys json))
    (terpri)
    (cond 
        ((null keys) json)
        ((json-p json)
            (let ((key (car keys)))
                (if (numberp key)
                    (get-json-value (cdr keys) (nth key json))
                    (let ((property 
                            (find-if 
                                #'(lambda (property) 
                                        (and (property-p property) 
                                            (eq (get-property-key property) (car keys)))) json)))
                        (if property
                            (get-json-value (cdr keys) (get-property-value property)))))))
        (t nil)))

For example, a json object produced by cl-json looks something like this:

Who Are in Space Now

25.04.2021 15:18

This weekend, I learnt how to fetch and parse JSON using the drakma HTTP client and cl-json libraries. And some simple error handling as well.

Tests:

(ql:quickload :testlib)

(use-package :testlib)

(run (context "who is in space?"
        (context "people-p"
            (context "when key is people" 
                (test "it should return true" (people-p (cons :people '()))))
            (context "when key is not people" 
                (test "it should return false" (not (people-p (cons :foo '()))))))
        (context "get-name"
            (context "when name is present" 
                (test "it should return name" (string= (get-name `(,(cons :name "foo"))) "foo")))
            (context "when name is not present" 
                (test "it should return nil" (eq nil (get-name `(,(cons :craft "bar")))))))
        (context "get-craft"
            (context "when craft is present" 
                (test "it should return craft" (string= (get-craft `(,(cons :craft "foo"))) "foo")))
            (context "when craft is not present" 
                (test "it should return nil" (eq nil (get-craft `(,(cons :name "bar")))))))
        (context "format-astronaut"
            (test "it should format correctly"
                (string= (format-astronaut `(,(cons :name "foo") ,(cons :craft "iss")) nil) (format nil "~&foo (iss)~%"))))
        (context "get-people"
            (test "it should return people list from property pair"
                    (equal (get-people (cons :people '(1 2 3))) '(1 2 3))))))

Implementation:

Csv Parser

19.04.2021 09:22

Writing a CSV parser is a good way to learn about streams and string formatting in Common Lisp. The goal is to take any CSV file and pretty print the contents. This is done by setting the width of each column to the widest value in that column + 1.

Code for the CSV parser:

(defun get-lines (input-stream)
    (let ((lines '()))
        (do ((line (read-line input-stream nil nil) 
                   (read-line input-stream nil nil)))
                   ((null line) (nreverse lines))
                   (push line lines))))

(defun get-columns (line)
        (let ((columns '()))
                (do ((i 0 (+ j 1)) 
                     (j (position #\, line :start 0) (position #\, line :start (+ j 1))))
                        ((null j) (progn (push (subseq line i) columns) (nreverse columns)))
                            (push (subseq line i j) columns))))
                            
(defun get-rows (lines)
    (map 'list #'get-columns lines))

(defun get-fitting-column-width (rows n)
    (let ((get-nth-column-width #'(lambda (row) (length (nth n row)))))
        (+ (apply #'max (map 'list get-nth-column-width rows)) 1)))

(defun get-fitting-column-widths (rows)
    (let ((cols (apply #'max (map 'list #'length rows))))
            (if (zerop cols) 
                '()
                (loop as n from 0 to (- cols 1) collect (get-fitting-column-width rows n)))))

(defun format-rows (rows &optional (output-stream t))
    (let* ((widths (get-fitting-column-widths rows))
           (format-row 
             (lambda (row) 
                (let ((n 0))
                    (map 'list 
                        #'(lambda (col)
                            (let ((w (nth n widths)))
                                (incf n)
                                (format nil (format nil "~~~AA" w) col))) row)))))
        (format output-stream "~{~&~{~A~}~%~}" (map 'list format-row rows))))

(defun format-csv (input-stream &optional (output-stream t))
    (format-rows (get-rows (get-lines input-stream)) output-stream))

And the test output of tests written using the little test library I have been refining:

Test Library Fixes

17.04.2021 16:56

The previous version of test library had a number of problems, most importantly it relies heavily on the lexical variables indent and outcomes shared across multiple test runs. This meant that if test execution was terminated halfway and restarted for a set of tests, the indent and outcomes might not have been correctly reset, hence resulting wrong test output formatting.

The fixed version:

(defvar *test-output-stream* t)

(defun make-spaces (n)
    (make-string n :initial-element #\space))

(defun format-outcome (depth name outcome &optional (output-stream *test-output-stream*))
    (format output-stream "~&~ATest `~A` ~:[failed~;passed~].~%" (make-spaces depth) name outcome))

(defun format-context (depth name &optional (output-stream *test-output-stream*))
    (format output-stream "~&~A~A~%" (make-spaces depth) name))

(defun format-outcomes (outcomes &optional (output-stream *test-output-stream*))
    (format output-stream "~&~A out of ~A tests passed~%" (count t outcomes) (length outcomes)))

(defmacro test (name predicate)
    (let ((outcome (gensym)))
        `(lambda (depth)
            (let ((,outcome ,predicate))
                (format-outcome depth ,name ,outcome) (list ,outcome)))))

(defmacro context (name &rest body)
    `(lambda (depth)
        (format-context depth ,name)
        (let ((depth (+ depth 1)))
           (reduce #'append (map 'list #'(lambda (f) (apply f `(,depth))) (list ,@body))))))

(defun run (root-context)
    (format-outcomes (apply root-context `(,0))))

The newly fixed version removes dependency on shared indent and outcomes lexical variables, instead explicitly redefining a depth lexical variable and passing in depth to each test and context, hence leveraging the exit of corresponding lexical scopes to restore the correct depth to use for test output formatting. An outcome is returned from each test and context, resulting in an accumulated list of outcomes at the root level. A run function is also introduced to kickstart a root context, henceforth removing the need for checking depth to determine if outcomes should be printed.

Previous Page 2 of 3 Next Page