gerard.au

Magic Mirror Update - Tram Tracker

Fri, 22 Aug 2025 00:00:00 +0000

As an extension to my MagicMirror project, I decided to add a tram tracker to lazily assist my communte to the nearest train station. Years ago, I had a LIFX globe that would flash when the tram was about to arrive, but I haven’t used it in a while. I did however, still have the code floating around on my private Bitbucket instance. The finest Node 6.10 code around.

The code for the light was based off the excellent, and apparently still running TramTracker SOAP service. Even way back in 2018 when I put this together, it was a bit of an oddity to use SOAP from a Node application, but with a few upgrades here and there to the various libraries, the thing roared back to life. I have no idea how I got a license key for this service originally, but it’s still up, and it’s still mostly working. You can’t seem to get the type of Tram any more from it (booooooo), but the arrival times are still there.

I ended up using Dennis Rosenbaum’s excellent MagicMirror template to load my new module up into MagicMirror. Effectively, the module has code which runs on the frontend, and then a mechanism to have code run on the backend (see node_helper.js), with an event-based mechanism for communication between them. If the frontend reloads, or a 30 second timer fired on the backend, then the new arrival information is sent down to the display.

The finished product looks like this. Nice and simple, with terrible colours and alignment, but I sort of love it. Even has the old Met logo.

MagicMirror Tram Arrival Plugin

Mirror Mirror on the Wall

Sun, 16 Mar 2025 00:00:00 +0000

With a young family, there is a lot of stuff going on at home. School, Kinder, work, various appointments, social outings. My wife and I run a shared calendar on Google, which is handy to have on our phones, but difficult to share with others, especially the kids. She really liked the look of the SkyCalendar, but it’s a bit expensive for a single use device (ie, a Calendar), and it also includes a monthly subscription fee for the majority of the features. So, what could I cobble together for a bit less?

The Hardware

I started off looking for some kind of large Android tablet as a dashboard display. Unfortunately in that 11inch+ size, the prices really start to escalate. What you can get, which are reasonably priced, are these small, generic, high resolution portable monitors from Alibaba or Amazon. It’s a 15.6 inch 1080P FHD Touch Monitor, with either USB-C or HDMI input. The screen itself is surprisingly bright on the highest setting, and definitely sharp enough for our needs. I’ve had an old Raspberry Pi 3 in the cupboard, which I decided would be perfect to drive it, but it only has HDMI out. You are supposed to be able to get the touch functionality working with a separate USB-A cable (which came in the box), but I couldn’t get it working with Raspbian. So, no touch for the first iteration.

I swung by Ikea, and picked up one of these Roedalm Birch frames to mount the screen in, which was roughly the right size.

MVP Picture

I really ruined the matt board within the frame, hastily cutting it out with a pair of scissors. Also, the length of the cable adaptors didn’t align correctly by having the frame centred. I have picked up a separate matt board, which I’ll cut with a bit more precision at a later date. MVP baby.

MagicMirror in Ikea Frame

The Software

I’ve had my eye on MagicMirror for ages, and although initially targeted at dashboards behind mirrors, it can be used for any kind of interactive display. I deployed it in Docker to my home server (an upcycled old MacBook Pro with a smashed display). MagicMirror does come with a bunch of included widgets, but the power is in the enormous ecosystem of plugins. Coming from the HomeBridge ecosystem, I found MagicMirror a bit more effort to get going. Plugins are effectively git cloned into a mounted directory (i’m running it in Docker on another machine), and then most plugins will have a NodeJS build step as well. Once the plugin is installed, you can then drop it in your config.json file, for it to appear on your dashboard. I ended up installing the following:

MMM-CalendarExt3Journal: The hero piece of the dashboard. Provides a weekly calendar view of the main calendar.
MMM-CalendarExt3Agenda: Provides a breakdown of events for the day. Handing to have a list of tasks in a consumable format.
MMM-Wallpaper: For cycling through a selection of images. This plugin has a variety of different image sourcing options, but I ended up just pointing it to a local directory.
MMM-Remote-Control: A plugin for remotely refreshing the dashboard. Perfect for the remotely managed raspberry pi.
MMM-auto-refresh: A plugin for refreshing the display.

Once you’ve got your widgets all connected, you then need to style it. There doesn’t appear to be the concept of themes within the core framework. You’re effectively forced to break open your browsers dev tools, to work out the various CSS selectors to start styling your dashboard. It’s certainly super customisable, but it is limited to how familiar you are with CSS. This is not for the feint hearted.

What it looks like in the end. Don’t just my colour choices.

MagicMirror

Sanitised example of my configuration file:

let config = {
    address: "0.0.0.0",	// Address to listen on, can be:
    port: 8080,
    basePath: "/",
    ipWhitelist: [],	// Set [] to allow all IP addresses
    useHttps: false,			// Support HTTPS or not, default "false" will use HTTP
    httpsPrivateKey: "",	// HTTPS private key path, only require when useHttps is true
    httpsCertificate: "",	// HTTPS Certificate path, only require when useHttps is true

    language: "en",
    locale: "en-US",
    logLevel: ["INFO", "LOG", "WARN", "ERROR"], // Add "DEBUG" for even more logging
    timeFormat: 24,
    units: "metric",
    modules: [
        {
            module: 'MMM-auto-refresh',
            config: {
                refreshInterval: 1800000
            }
        },
        {
            module: "clock",
            position: "top_left", // This can be any of the regions.
            config: {
                timeFormat: "12",
                displaySeconds: false
            },
        },
        {
            module: "calendar",
            config: {
                calendars: [
                    {
                        broadcastPastEvents: true,
                        name: "family",
                        color: "green",
                        fetchInterval: 5 * 60 * 1000,
                        maximumEntries: 10000,
                        url: "https://calendar.google.com/calendar/ical/XXXX" // Shared Google Calender
                    },
                    {
                        broadcastPastEvents: true,
                        name: "uniforms",
                        color: "blue",
                        fetchInterval: 24 * 60 * 60 * 1000,
                        maximumEntries: 10000,
                        url: "https://calendar.google.com/calendar/ical/XXXXX" // School Uniform Calendar
                    },
                    {
                        broadcastPastEvents: true,
                        name: "food",
                        color: "yellow",
                        symbol: "utensils",
                        fetchInterval: 5 * 60 * 1000,
                        maximumEntries: 20,
                        url: "http://192.168.1.193:13000/calendar" // Meal Plan generated from Apple Note
                    },
                ]
            }
        },
        {
            module: "MMM-CalendarExt3Agenda",
            position: "top_right",
            config: {
                endDayIndex: 1,
                showMiniMonthCalendar: false
            }
        },
        {
            module: "MMM-CalendarExt3Journal",
            position: "bottom_bar",
            config: {
                height: '68vh', // Half of the screen
                hourLength: 16, // Display 12 hours
                beginHour: 6, // From 7 o'clock
                days: 7, // For 7 days,
                locale: "en-AU",
                calendarSet: ["family", "uniforms", "food"]
            }
        },
        {
            module: "MMM-Wallpaper",
            position: "fullscreen_below",
            config: {
                source: "local:/pictures",
                slideInterval: 10 * 60 * 1000 // Change slides every minute
            }
        },
        {
            module: 'MMM-Remote-Control',
            config: {
                customCommand: {},  // Optional, See "Using Custom Commands" below
                showModuleApiMenu: false, // Optional, Enable the Module Controls menu
                secureEndpoints: false, // Optional, See API/README.md
            }
        },
    ]
};

/*************** DO NOT EDIT THE LINE BELOW ***************/
if (typeof module !== "undefined") { module.exports = config; }

Meal Planning with Apple Notes

My wife’s other main requirements, was being able to show what the week’s meal plan situation looks like. I do a lot of the supermarket shopping and cooking, but it’s important to have visibility of what we’re having across the week. A meal plan is something that doesn’t naturally fit very well within a calendar, and is something I usually put together in Apple Notes. So, how do I get my Apple Notes based data onto my MagicMirror? I was initially fretting about having to write some kind of iOS based application, to interact with Notes.

After going down a rabbit hole, Notes doesn’t appear to be accessible via 3rd party apps. However, what you can integrate with is iOS’s Shortcuts. You can use it to run a small script after the application is minimised. What I’ve done, is effectively extract the contents of the Dinner note every time the application is closed (and I’m on my home network), and make a HTTP Post call to a specific endpoint. I’m amazed that this something Apple supports out of the box.

Dinner Selection Note

Shortcuts Example

Then, mainly using Github Copilot, I wrote a small application which parses the Monday-Sunday header separated Notes file, and then serves the data within the ICS Calendar format, so the standard Calendar module can parse the data. The results?

Meal Planning

Turning the Display On/Off

The power of home based automation comes when you can tie all your bits and bobs together. I’m running a Zigbee2MQTT with HomeBridge setup, for control predominately via Apple Homekit, with complex automations handled by NodeRed. I’ve got a couple of IR motion Zigbee sensors within the room (think Philips Hue Motion Sensor) which coupled with NodeRed, allow you to run automations based on motion. With the motion data available, I’m able to then power off the display when nobody is present within the room; saving power when nothing is actually looking at it.

NodeRed Display Control

I’ve loaded a plugin into NodeRed to allow it to trigger shell scripts on the RaspberryPi, which then turn the display on/off when called.

#!/bin/bash

if [ "$1" == "off" ]; then
  wlr-randr --output HDMI-A-1 --off
elif [ "$1" == "on" ]; then
  wlr-randr --output HDMI-A-1 --on --mode 1920x1080
else
  echo "Invalid argument. Use 'off' or 'on'."
  exit 1
fi

Fin

It’s been incredibly fun building something IoT based which is actually solving a problem for my family. So many moving parts, but I think the result is worth it in the end. I can’t wait to explore some additional plugins, to make the display even more useful. Onwards and upwards.

Frolicking with AWS Bedrock with Knowledge Bases

Fri, 14 Feb 2025 00:00:00 +0000

Frolicking with AWS Bedrock with Knowledge Bases

I’ve been meaning to play with AWS Bedrock’s Knowledge Bases feature for a while now. The ability to load relevant organizational data, and have it available through an AWS-powered service, is very appealing. Many enterprise customers are reluctant to run anything more homegrown, like Ollama, but being able to point at an AWS service and say that it does a thing, is a far easier sell than spinning up something yourself, that you then need to maintain.

For those using Bedrock for the first time, you actually need to request individual model access through the Console, which confuses me; I prefer it when everything in AWS land just works out of the box. I understand why you might need to request access for a third-party model, but the Amazon ones should just work out of the box. Anyone who has tried to use a new AWS region knows what I’m talking about; I’m looking at you ap-southeast-4. Anyway, I’ve added the latest Anthropic and the Titan Text Embeddings V2 (you’ll see why shortly) models in the screenshot below. Get yourself a cup of coffee, and hopefully, they’ll be enabled when you return.

![][/images/model_list.webp]

Once you’ve been granted keys to the kingdom, it’s time to look at the Knowledge Bases feature. Knowledge Bases have effectively two main purposes. Firstly, to ETL data from a data source (S3, WebCrawler, Custom API) into a Vector Store. Secondly, to surface that Vector Store data is part of a Retrieval-Augmented Generation (RAG) process when interacting with your targeted model.

The RAG process helps to improve accuracy, and relevance of text returned by the model, by providing data that is (hopefully) factually accurate so it can provide a tailored response. Think of it like this; the underlying model knows a bunch of general facts that have been wired into it as part of its training, but it knows nothing about the specifics of your organisation. For example, your organisation might use specific acronyms for project names. By providing those acronym references as part of a RAG, the model will be able to respond with those org-specific acronyms.

![][/images/provide_knowledge.webp]

The AWS Console gives a great overview of the parts you need to initialize a Knowledge Base, but I prefer to assemble it myself via AWS’ Cloud Development Kit (CDK). It gives you a greater appreciation for what actually is required to get it running, and doesn’t abstract away what it’s doing in the background. It also makes it much easier to spin up and down resources, which is a great way of reducing costs during development. For example, there is no point leaving things running overnight, when nobody is using them.

The CDK constructs at the time of writing (2024-12-03), were only Level 1, meaning you need to hand-wire in all the dependencies yourself. Not only do you need to wire in the dependencies yourself, but you also need to provide a mechanism to load up the index into the OpenSearch Vector database. This was all getting a little too hard.

![][cdk_example.webp]

Luckily, Lee James Gilmore has an excellent post describing the use of Level 3 Construct from AWS Labs. No more crying in CDK, and the ability to deploy and destroy a Knowledge Base stack, relatively easily. Although it does hide away some of what it’s doing with the token parsing, it’s all there on your machine for you to look at, instead of hiding throughout the Console.

import \* as s3 from "aws-cdk-lib/aws-s3";  
import { bedrock } from "@cdklabs/generative-ai-cdk-constructs";

const kb \= new bedrock.KnowledgeBase(this, "KnowledgeBase", {  
embeddingsModel: bedrock.BedrockFoundationModel.TITAN\_EMBED\_TEXT\_V2\_1024,  
instruction: "Use this knowledge base to answer questions about books. " \+ "It contains the full text of novels.",  
});

const docBucket \= new s3.Bucket(this, "DocBucket");

new bedrock.S3DataSource(this, "DataSource", {  
bucket: docBucket,  
knowledgeBase: kb,  
dataSourceName: "books",  
chunkingStrategy: bedrock.ChunkingStrategy.fixedSize({  
maxTokens: 500,  
overlapPercentage: 20,  
}),  
});

	const  crawlDataSource \= new bedrock.WebCrawlerDataSource(this, "IpponWebsite", {  
  	knowledgeBase: kb,  
  	crawlingRate: 10,  
  	sourceUrls: \["https://www.ipponaustralia.com.au/"\],  
  	chunkingStrategy: bedrock.ChunkingStrategy.fixedSize({  
    	maxTokens: 500,  
    	overlapPercentage: 20,  
  	}),  
	})

You’ll notice a couple of things with this CDK snippet:

I’ve added the TITAN\_EMBED\_TEXT\_V2\_1024 as the embedding model. This is used as part of the data parsing process, and must be enabled via the model selector.
There are two data sources for our knowledge base. There is an S3 and amazingly an inbuilt Web-Crawler option.

Once you’ve deployed your Knowledge Base, you can load some files into your target S3 bucket, for your Knowledge Base. I uploaded some PDFs of our fancy new practice decks to see how it would go with those. It supports a wide range of other document formats, but would suggest probably using something that isn’t a PDF in more of a production setting.

Data is not automatically pulled into your Knowledge Base, you need to sync it. The Console or the CLI can do that for you.
![][images/data_source.webp]

Once the data has been loaded, you can test the model via the Console Test Knowledge Base feature, once you specify a model.

![][images/chat_example.webp]

Then, ask away via the Chatbot interface. You can see it pulling information from the documents we’ve loaded.

![][images/ippon_chat_example.webp]
The test interface is good for getting a feel for the kind of information it can extract. If you turn off the Generate Responses toggle, you can see the kinds of information extracted from the Vector Search database. The Vector Search db contains effectively snippets of text, which is then passed on to the LLM. The formatting in my PDFs was a little off (I printed them from Google Slides), but I expect it would be better with a text-first medium such as Confluence. It was easy to have it give long-winded answers, suggesting it needed help with the formatting. Simple questions, like How many people work at Ippon?, could be easily extracted.

![][images/generate_responses.webp]

But how do I use it?

Although logging onto the Console is super fun, chat-based interfaces need to be accessible to be useful. Slack is a great avenue for that integration, and there is no shortage of tutorials from AWS themselves on integrating Slack Chatbots. But what if we got Claude 3.5 Sonnet to “write” a quick and dirty Lambda function for us?

Here’s the final prompt I ended up with.

Write me a NodeJS AWS Lambda function to handle Slack messages. Have it access all Secrets via the AWS Secret Manager and verify that the inbound Slack request is valid. Once it receives a message, it sends it to AWS Bedrock, using the `Claude 3.5` model, and using a Knowledge base called: XXXX KnowledgeBaseId.

![][images/generate_node_program.webp]

Although the code it generated looked plausibly correct at first glance, it had some issues.

The first version didn’t validate the inbound Slack token at all (it just accepted it) and ignored the outbound token. Correcting it via prompts made it fix the issue.
It mixed up the body parsing of incoming requests. Instead of parsing the JSON body, it tried to parse it as a query string.
The expected body from Slack wasn’t exactly what it was expecting so that needed to be massaged.
It defaulted to using an older NodeJS version and AWS SDK, which won’t do.
Slack gets unhappy if you don’t respond within 3 seconds to their requests. Due to startup speeds, it’s sometimes unable to respond in time. I ended up using response streaming and swapping out the implementation suggested from the model.

I hammered it into something that looks like the following (⅔ written by AI, with a lot of debugging by me):

const { SecretsManagerClient, GetSecretValueCommand } \= require("@aws-sdk/client-secrets-manager");  
const { RetrieveAndGenerateCommand, BedrockAgentRuntimeClient } \= require('@aws-sdk/client-bedrock-agent-runtime')  
const crypto \= require('crypto');

const secretsManager \= new SecretsManagerClient({ region: "us-east-1" }); // Replace with your region  
const bedrockRuntime \= new BedrockAgentRuntimeClient({ region: "us-east-1" }); // Replace with your region

let SLACK\_SIGNING\_SECRET;  
let SLACK\_BOT\_TOKEN;

async function getSecrets() {  
if(SLACK\_SIGNING\_SECRET && SLACK\_BOT\_TOKEN) {  
return;  
}  
try {  
const secretName \= "slack-bedrock-secrets";  
const response \= await secretsManager.send(new GetSecretValueCommand({ SecretId: secretName }));  
const secrets \= JSON.parse(response.SecretString);  
SLACK\_SIGNING\_SECRET \= secrets.SLACK\_SIGNING\_SECRET;  
SLACK\_BOT\_TOKEN \= secrets.SLACK\_BOT\_TOKEN;  
} catch (error) {  
console.error("Error retrieving secrets:", error);  
throw error;  
}  
}

function verifySlackRequest(event) {  
const slackSignature \= event.headers\['x-slack-signature'\];  
const timestamp \= event.headers\['x-slack-request-timestamp'\];  
const body \= event.body;

const baseString \= `v0:${timestamp}:${body}`;  
const hmac \= crypto.createHmac('sha256', SLACK\_SIGNING\_SECRET);  
const computedSignature \= `v0=${hmac.update(baseString).digest('hex')}`;

return crypto.timingSafeEqual(Buffer.from(computedSignature), Buffer.from(slackSignature));  
}

async function invokeBedrockModel(prompt) {  
const retrieveAndGen \= new RetrieveAndGenerateCommand({  
input: { text: prompt },  
retrieveAndGenerateConfiguration: {  
type: "KNOWLEDGE\_BASE",  
knowledgeBaseConfiguration: {  
knowledgeBaseId: process.env.KNOWLEDGE\_BASE\_ID,  
modelArn: process.env.MODEL\_ARN,  
}  
}  
})

try {  
const response \= await bedrockRuntime.send(retrieveAndGen);  
return response.output.text  
} catch (error) {  
console.error("Error invoking Bedrock model:", error);  
throw error;  
}  
}

async function respondToSlack(channel, text) {  
console.log("Responding to slack: ", channel, ", text:", text)  
const response \= await fetch('https://slack.com/api/chat.postMessage', {  
method: 'POST',  
headers: {  
'Authorization': `Bearer ${SLACK\_BOT\_TOKEN}`,  
'Content-Type': 'application/json'  
},  
body: JSON.stringify({ channel, text })  
});

if (\!response.ok) {  
throw new Error(`Failed to send message to Slack: ${response.statusText}`);  
}  
}

function writeStreamResponse(responseStream, statusCode, body) {  
const httpResponseMetadata \= {  
statusCode: statusCode,  
headers: {  
},  
};  
responseStream \= awslambda.HttpResponseStream.from(  
responseStream,  
httpResponseMetadata  
);  
responseStream.write(body);  
responseStream.end();  
}

exports.handler \= awslambda.streamifyResponse(async (event, responseStream, \_context) \=\> {  
try {  
await getSecrets();

	if (\!verifySlackRequest(event)) {  
  	writeStreamResponse(responseStream, 401, "Unauthorized" )  
  	return {};  
	}

	const body \= JSON.parse(event.body);  
	console.log("Request: ", body)  
	const {channel, text, bot\_profile} \= body.event;

	if (body.type \=== 'url\_verification') {  
  	writeStreamResponse(responseStream, 200, body.challenge )  
  	return {};  
	}  
	// This check is a big hamfisted. Ideally, it should check if the message is coming from the bot, and then ignore  
	// but this works for a PoC.  
	if (body.type \=== 'event\_callback' && body.event.type \=== 'message' && \!bot\_profile) {  
  	console.log("\[Function\] Returning response to client");  
  	writeStreamResponse(responseStream, 200, {})

  	const aiResponse \= await invokeBedrockModel(text);  
  	console.log("Generated response: ", aiResponse)  
  	await respondToSlack(channel, aiResponse);  
	}  
	return {};  
} catch (error) {  
console.error("Error processing request:", error);  
console.dir(error);  
writeStreamResponse(responseStream, 500,  "Internal Server Error")  
return {};  
}  
});

This is an incredibly simple request/response demo and doesn’t deal with any of the standard conversational features you’d need for an actual working chatbot. It doesn’t maintain any context between the Slack conversation and the Claude calls, so it effectively “forgets” what you’ve asked. You’d definitely need to look into storing that data in something like DynamoDB to maintain the context between event calls from Slack. For a simple question/answer approach though, it works well. At one point, I got it stuck in a loop while it was responding to itself. Watch it burn those tokens (Don’t tell Duncan).

![][bot_error.webp]

Actual success with the code above! I generally will try to iterate on the code within the LLM chat for as long as possible, because it makes it much harder to partially debug portions of it if you’re constantly needing to manually merge any changes you make with the LLM output.
![][images/sucess_message.webp]

In Conclusion

Once you get your head around the concepts, and the various parts that need to be glued together, the AWS Bedrock offering, with Knowledge Base, is an incredibly interesting proposition, and a great way to start integrating these very powerful tools into your products. As with a lot of these AI offerings, it’s important not to slap it on everything just to see what sticks. Have fun, and stay safe out there.

OpenTelemetry and Friends @ Programmable

Fri, 03 Mar 2023 02:01:02 +0000

I was lucky enough to present at Programmable, the Australian software engineering conference on OpenTelemetry.

I’ve uploaded a copy of my presentation here, as well as the demo source code for the Microservices example. As you can see, my frontend development skills are top notch.

OpenTelemetry and Friends

Tue, 15 Nov 2022 14:01:02 +0000

What is OpenTelemetry

OpenTelemetry (OTel) is an open-source initiative to provide a standardised approach for the capture and distribution of metrics, trace and log data from applications. It defines not just APIs and schemas, but also a set of standard vendor-neutral SDKs and monitoring agents to facilitate collection and exports.

The documentation itself is very detailed, if a little overwhelming in parts. The core to delivering applications that utilise OTel is the OpenTelemetry Collector. The OTel Collector is a small standalone executable that can be run as a sidecar in a container environment, or as a layer if used with something like AWS Lambda.

The OTel Collector, at a high level, can be divided into three main parts:

Receivers: Can, via endpoints, receive data from applications, for example using otlpreceiver, or they can be configured in a pull mode to scrape data from an application, for example using the prometheusreceiver receiver. I
Processors: Can bundle/modify the metric/trace/log data as it comes through. The common batchprocessor groups and compresses data together to reduce load on exporters.
Exporters: Exporters take the metric/trace/log data and export it to a downstream metric/trace/log solution. So for example, you might want to have trace data analysed with AWS X-ray, but have log data go to Splunk. In the diagram above, we have metric data going to AWS CloudWatch, using the awsemf exporter.

Arise the OTel Distro

The OTel project released their own standard collector, which includes a bunch of sensible default receivers and exporters. However, what appears to be have happened is that a number of vendors have released expanded distributions of the collector, with additional bundled receivers and exporters to align with their target products. For example AWS has their AWS Distro For Open Telemetry which includes receivers for various AWS container technologies, and exporters for X-ray and CloudWatch. SumoLogic has a distribution which supports exporting to SumoLogic (for both metrics and tracing). These specific distros are great for getting going quickly within a given ecosystem, but can cause issues if you want to route data to multiple providers.

Other providers, such as NewRelic and Honeycomb.io provide HTTP/gRPC endpoints that can directly receive data from the standard otlpexporter exporter, so you can just use the standard collector.

Agents, SDKs, Existing Libraries Oh-My

Now that you understand how the collector works, the next step is getting your valuable application data into the collector. This is going to very much depend on the language your application is written in. In Java land, besides a manual SDK, there is support for an agent-based automatic connector, which out of the box has a lot of support for popular Java libraries. If you would rather not use an agent-based approach, the OpenTelemetry project has an alternative mechanism for injecting trace behaviour specifically for SpringBoot.

SpringBoot doesn’t yet support OpenTelemetry natively out of the box, but does support a number of metric export mechanisms, including Prometheus. In this case, the collector can be configured with a prometheus receiver on the collector, which will scrape metrics from SpringBoot at a defined interval.

If you’re looking into Quarkus, it supports only tracing currently.

How do I use it with AWS?

AWS has some detailed documentation around integrating with their various services, with X-Ray being the target for tracing information. A common deployment pattern is using the collector as an ECS side-car, which will accept tracing data generated from a primary app, and then diseminate it to X-Ray for evaluation.

The following is a sample CDK pattern, which deploys a Quarkus app (as described in their documentation), into ECS.

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import {aws_ec2, aws_ecr_assets, aws_ecs, aws_ecs_patterns, aws_iam} from "aws-cdk-lib";
import {DockerImageAsset, NetworkMode} from "aws-cdk-lib/aws-ecr-assets";
import * as path from "path";
import {ContainerImage} from "aws-cdk-lib/aws-ecs";

export class OtelDemoStack extends cdk.Stack {
    constructor(scope: Construct, id: string, props?: cdk.StackProps) {
        super(scope, id, props);

        const defaultVPC = aws_ec2.Vpc.fromLookup(this, 'ImportVPC',{isDefault: true});

        const cluster = new aws_ecs.Cluster(this, "OtelCluster", {
            vpc: defaultVPC
        });

        //
        const dockerImageServiceB = new DockerImageAsset(this, 'OtelBuildServiceB', {
            directory: path.join(__dirname, '../','../','service_b'),
            networkMode: NetworkMode.HOST,
            file: 'src/main/docker/Dockerfile.jvm',
            platform: aws_ecr_assets.Platform.LINUX_AMD64,
            ignoreMode: undefined,
        })

        const otelPolicy = new aws_iam.PolicyDocument({
            statements: [new aws_iam.PolicyStatement({
                actions: [
                    'logs:PutLogEvents',
                    'logs:CreateLogGroup',
                    'logs:CreateLogStream',
                    'logs:DescribeLogStreams',
                    'logs:DescribeLogGroups',
                    'xray:PutTraceSegments',
                    'xray:PutTelemetryRecords',
                    'xray:GetSamplingRules',
                    'xray:GetSamplingTargets',
                    'xray:GetSamplingStatisticSummaries'
                ],
                resources: ['*'],
            })],
        });

        const taskRole = new aws_iam.Role(this, 'Role', {
            assumedBy: new aws_iam.ServicePrincipal('ecs-tasks.amazonaws.com'),
            inlinePolicies: {
                otelPolicy
            }
        });

        const fargateTaskDefinition = new aws_ecs.FargateTaskDefinition(this, 'TaskDef', {
            memoryLimitMiB: 512,
            cpu: 256,
            taskRole: taskRole
        });
        let webcontainer = fargateTaskDefinition.addContainer("WebContainer", {
            image: ContainerImage.fromDockerImageAsset(dockerImageServiceB),
            logging: new aws_ecs.AwsLogDriver({ streamPrefix: 'ServiceB', mode: aws_ecs.AwsLogDriverMode.NON_BLOCKING })
        });
        webcontainer.addPortMappings({
            containerPort: 8080,
        });
        fargateTaskDefinition.addContainer("Otel", {
            image: aws_ecs.ContainerImage.fromRegistry('amazon/aws-otel-collector:latest'),
            logging: new aws_ecs.AwsLogDriver({ streamPrefix: 'ServiceBOtel', mode: aws_ecs.AwsLogDriverMode.NON_BLOCKING }),
            command: ['--config=/etc/ecs/ecs-default-config.yaml', '--set=service.telemetry.logs.level=DEBUG']
        });

        // // Create a load-balanced Fargate service and make it public
        let serviceB = new aws_ecs_patterns.ApplicationLoadBalancedFargateService(this, "SampleOtelService", {
            cluster: cluster,
            cpu: 512,
            desiredCount: 1,
            taskDefinition: fargateTaskDefinition,
            assignPublicIp: true, // this is purely to not require a NAT option.
            memoryLimitMiB: 1024,
            publicLoadBalancer: true,
            healthCheckGracePeriod: cdk.Duration.seconds(10),
        });
        dockerImageServiceB.repository.grantPull(serviceB.taskDefinition.obtainExecutionRole());

    }
}

It’s important to note, that X-ray currently requires a specific time based ID format, which Quarkus discusses in this section. You effectively need to use the X-Ray IDGenerator, or nothing will appear in the X-ray console.

The X-ray console looks like the following. This example demonstrates hitting a sample /hello endpoint, via the deployed load-balancer, hitting the Quarkus application, and then making a HTTP GET call to a Star Wars test endpoint: https://www.swapi.tech/api/starships/3.

Why/When Would I Use It?

It’s a great way of integrating metric and trace monitoring into new applications, or applications that don’t have existing solutions in this space. I’ve excluded logging for the moment as the specification is still in a draft status at the time of writing, but it is certainly an exciting development. This is a pretty fast moving space, which will mature and solidify over time, but it will be the way of integrating vendor-agnostic tracing/metric/logging functionality into your application moving forward.

Conclusion

If you’re starting a new project, or retrofitting metric and tracing support to an existing application, defintiely look into OpenTelemetry. It provides a solid, standards compliant mechanism to surface application telemetry, and works well even if you haven’t yet decided on your downstream vendors as yet. Stop waiting and start exporting!

Lambda Docker Adventures with SQLite

Wed, 12 May 2021 00:00:08 -1000

Last year I spent some time playing around using Tegola (a mapping tile generator) and AWS Lambda to generate map tiles using Geospatial data stored in SQLite and hosted on Amazon EFS. There was a serious performance penalty reading large data files over EFS, so I needed to make sure my database was as small as possible. In addition, you pay for all that data that you’re transferring, so although not expensive for a low volume site, you could end up in a pickle if things pick up.

To further that thread of SQLite data in Lambda, in December last year as an early Christmas present AWS released Container Image support, a sort of half-way between a traditional Lambda Zip function and a ECS-Fargate container. This allows you to generate a Docker image locally, push it into the Elastic Container Registry (ECR), and then use that as a Lambda function. Cool, but what really caught my eye was the fact that it supports up to a 10Gig container image. So, was using Docker+Lambda a better solution for read-only SQLite databases than Lambda+EFS? I wanted to build out something practical to validate the technique, so I decided on an address auto-complete endpoint, where it turns free text into decomposed address data. For example, if you supply “101 Coll” as text, I want it to return the decomposed address information for “101 Collins Street”.

G-NAF Fun with SQLite

The data I was using last time originated from the Australian Government’s Geoscape Geocoded National Address File (G-NAF), which contains a record of every single physical address in Australia. The data itself is provided in a pipe-seperated set of normalized files, which are easily loadable into Postgres/Postgis with the included set of table schemas; once you get around the table structure.

Now I wanted to load this address data into SQLite, so I needed to make a couple of less-than-ideal modifications to the standard load process:

Schema Refactor: SQLite needs its foreign keys defined upfront, so I needed to modify the included G-NAF schema for that.
Null Imports: This took me far longer than I care to admit to fix(and I only identified it as an issue later), but there is currently no mechanism to have SQLite’s default CSV import mechanism to treat blank values as nulls. So my database import was originally littered with empty char fields. I ended up using a blunt hammer to update all empty strings to NULL through a set of SQL UPDATE commands; not pretty, but it works. Also very slowwww.

The end product is this enormous Docker Image (https://hub.docker.com/repository/docker/ggotti/g-naf-full-sqlite), based off this Github repository. It’s a great way of being able to quickly see what G-NAF has to offer, without needing to fumble around with the schema.

Distilling it Down

With my frankenstein SQLite image, it was time to convert that into something I could use in my Lambda function. I created a new SQLite database, and then I materialised the “ADDRESS_VIEW” from the G-NAF schema into a separate table in my new database, which is effectively a denormalized view of the address data.

The next step was to create a text representation of the address data. The excellent team at ISP Superloop have come up with a MySQL version of converting the data into text. Unfortunately, it makes extensive use of `CONCAT_WS`, which again is something not supported by SQLite. What I ended up doing was hand converting their statement into an absolutely unmaintainable mess of a thing, which sort of replicates the `CONCAT_WS` behaviour using syntax that’s available. It’s not 100%, but it’s close enough. Look at this horrific mess. Warning. May Burn your Eyes Out.

AWS SAM supports the build and development of Docker Images as part of the CLI. I ended up using that to build the project scaffold, and then introduced a multi-step Docker Build process to pull in the SQLite image, generate the distilled address database, and then copying the generated database into the appropriate NodeJS Lambda docker image: public.ecr.aws/lambda/nodejs:14. I was originally building this on my 2015 Macbook Pro, and it was seriously struggling with my grotesquely unoptimised build process. I ended up switching it to an older Linux desktop, but it didn’t struggle as much with the heavy disk IO.

SAM itself is really getting better and better with every release. It’s so quick to just spin up a new application. The only thing I had to do outside of SAM was define the Elastic Container Repository (ECR), and just feed those details into SAM. sam build && sam deploy and you’re good to go.

Demo

The Lambda application itself is pretty simple, with a small JavaScript application connecting to SQLite and doing a LIKE search against one of the tables. This could probably be improved with the FTS5 Full Text Search extension, but LIKE works well enough for this demo. To demo it’s usage, I put a simple web-page using autoComplete.js to hit the Lambda function as the user types. Give it a try here, or watch the above video.

Source code can be found on Github here.

Revisiting Tegola

So the real question is, how does the Docker-Image approach work in comparison to reading large SQLite databases off EFS? The results may surprise you. I left the RAM the same (3008mb), and then ran a comparison between the two, against the set of same Tagola coordinates; a rather complex area of the Melbourne CBD. I manually adjusted the RAM between requests, to guarantee a fresh instance of the function. The Lambda first function was consistently taking 25–26 seconds to process the request. Whereas the Docker image was taking between 22–24 seconds to process, when it worked. A large number of requests were hitting the API Gateway’s 30 second limiter, and therefore timing out. There seemed to be a slight delay when the Image was being pulled for the first time, which may factor into the delays. I suspect behind the scenes it does some caching between ECR and Lambda of the image itself, independent of the actual Lambda execution “warmth” behaviour. Increasing the amount of available RAM to 4GB seemed to improve performance consistently.

So there seems to be a slight increase in the RAM requirements for the Docker image. However, the usability of the Docker format for pushing data in, and the fact that the ECR data transfers within the same region are free, make the Docker option pretty appealing.

Conclusion

So, can you use SQLite with Docker in Lambda? You can. Can you import G-NAF data into a relatively compact SQLite database? You can. So is it worth it in the end? The experience with SAM is pretty seamless, and it’s a great way of pushing large read only data images around; less than 10GB currently supported. It certainly removes some of the complexity of managing the data via EFS. Just be prepared to pay a little more on your execution costs running it via Docker.

PS: If you were fully crazy you could probably build some kind of updatable website where you have a separate Lambda function updating an ECR image based off a singular write data fed in via an SQS queue. Maybe.

Lambda Tile Generation with Tegola

Tue, 27 Oct 2020 00:00:08 -1000

Inspired by the excellent NBN MTM Alpha I’ve been recently playing around with the NBN connection type data released on data.gov.au in July 2020. Combining it with Geocoded National Address File , you can produce maps that surface address level connection details. For NBN enthusiasts (such as myself), it’s pretty exciting, as the actual NBN map is purposely evasive.

So once you’ve loaded all that juicy data into PostGIS how do you get that information out? The open source mapping space has changed significantly since I played around with it in the early 2010s. Tile generation no longer revolves around muttering incantations to summon life out of Mapnik for raster tile generation. Now, there is the MapBox Vector Specification which allows you to serve piping hot geo data into the complementary Mapbox GL JS mapping library. To serve vector tiles using the MapBox Vector Specification, Tegola caught my eye as a lightweight, well supported application. Most excitingly, it supports Lambda based generation.

The Idea

The majority of articles (such as Running a Serverless Vector Tile Backend with AWS Lambda and Go) talk about running Tegola with a Postgres backend, and the associated connection pool issues with that. If I was doing this at work, I’d be looking at potentially an Amazon RDS Proxy with a smallish RDS instance. Even on a db.t3.small RDS Postgres instance, you’d be looking at $25.92 for the proxy, and $40.32 for the database. That’s $66.24 USD a month, which is $93.15 Australian Dollarydoos. That’s too much money. I could run a LightSail/Digital Ocean instance, but I’d want to put a Load Balancer in front ($18 USD for a LightSail LB). So how can we run it on Lambda, but not need a database?

Enter Lambda’s support for Amazon Elastic File System (EFS) which was announced in June. No longer do you need to use nested layers, or some S3 tomfoolery to load data into the 512-MB temporary storage available to Lambda. So with an expandable filesystem, it really opens up the possibilities for making larger files available to Lambda.

Tegola primarily supported PostGIS, but it also supports GeoPackage, an SQLite based standard for storing geospatial data. Now, SQLite and NFS don’t really mix. But this will be read-only, so we’ll just give it a go. So Tegola for tile generation, with a GeoPackage on NFS for the data storage. What could go wrong?

The Reality

My initial plan was to run the NBN dataset, as well as an OSM extract of Australia. I loaded them Postgis using Tegola OSM, and once I had it working, I fumbled through using ogr2ogr to export the data from PostGis into GeoPackage. Make sure you use the correct spatial reference system (SRS) otherwise bad things will happen; Tegola prefers 4326 for its GeoPackage configurations. Some of the data types also caused issues with the conversion, so I commented out a number of the layers in the Tegola OSM config. All up, my OSM data was about 4gig, and my NBN data was about 3.5gig. So once I had it working on my local computer, it was time to load it into Lambda.

Tegola has a pre-built Lambda version, so that was simple, as was wiring in the EFS volume. Of note, I did need to use an EC2 instance to load my 7.5gig of data into EFS, which was unfortunate, as I was hoping I could use an S3-style console for uploading. The only struggle I had was getting the API Gateway setup, along with CORS, and that’s mainly because I didn’t read the documentation correctly about setting the `’application/octet-stream` media-type; this Github Issue also helped. So with that corrected, i wired it up to my web-app and I was good to go…except…

It didn’t work. I bumped the memory allocation. Still didn’t work. Looking into the logs, it appeared it was timing out during the execution; hitting the 30 second API Gateway timeout. Using EFS requires connectivity via your VPC, but recent improvements have drastically reduced those cold-start times. The biggest issue seemed to be the size of the GeoPackage files. Removing the dream of serving the OSM data myself, I left the NBN GeoPackage file, and it worked significantly better. It still choked on denser areas of the map, for example the Melbourne CBD, and for that I’ve reverted to using an S3 based cache mechanism. Tegola has an excellent cache seeding function, so data can be served instead from a cache. A word of warning on caching, it can be very expensive, both from a time + storage perspective, to cache tiles at lower zoom levels. Cache seed only where something is expensive to generate on the fly.

Component Overview

Conclusion

So, can you serve MapBox Vector Specification tiles on the cheap using Lambda + Tegola + EFS? Yes you can. But caveat emptor. You’ll need to make sure your GeoPackage aren’t too big, potentially splitting them into specific layers; instead of the giant single layer I used. You’ll also want to make liberal use of the cache-seeding function, where possible, but that does come with a rather large file size overhead.

Colliding with Quarkus

Tue, 10 Sep 2019 17:11:00 +0000

Recently at Voxxed Days Melbourne there was a Red Hat booth, and they were handing out Quarkus stickers. Obviously if it’s good enough to put on a sticker, it must be important. So, what is Quarkus? According to their website, Quarkus is a Cloud Native, (Linux) Container First framework for writing Java applications. It’s really a mix of two tightly integrated components: a Java Framework and GraalVM.

Java Framework

On first impressions, the framework seems very similar to SpringBoot, which isn’t a bad thing at all. At its core, it supports all the dependency injection features you could want, based on this. From there, Quarkus combines a list of curated extensions to allow you to easily bolt on new functionality as required. If you need to include a REST server, you can include the quarkus-resteasy dependency. If you need access to AWS’ DynamoDB, there is quarkus-amazon-dynamodb. You don’t need to go hunting for dependencies or versions to use; all the important stuff is available for you to pull in via the listExtensions command.

GraalVM

GraalVM is a relatively new universal virtual machine for running JVM-based languages, as well as JavaScript, Python, Ruby and R. The feature that Quarkus really embraces is the SubstrateVM, which allows Java applications to be compiled into native executable images. Instead of running your Java code against a JVM, you can compile it down to a portable, native executable. Quarkus abstracts all this complexity from you, and allows you to just run nativeImage to build a executable binary.

Taking it for a spin

So I have put together a small HTTP server example which basically just proxies requests to the https://api.kanye.rest service, which returns a random Kanye West quote. The application has three small classes:

VoteKanyeResource
KanyeService
KanyeQuote

VoteKanyeResource

The class uses dependency injection to inject the KanyeService, which is then invoked in the quote method, which is defined as HTTP accessible using the @GET annotation.

package au.com.gigliotti.quarkus;

import org.eclipse.microprofile.rest.client.inject.RestClient;

import javax.enterprise.context.ApplicationScoped;
import javax.inject.Inject;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.CompletionStage;

@Path("/quote")
@ApplicationScoped
public class VoteKanyeResource {
    @Inject
    @RestClient
    KanyeService quoteService;

    @GET
    @Produces(MediaType.TEXT_PLAIN)
    public CompletionStage<String> quote() {
        return CompletableFuture.supplyAsync(() -> quoteService.getRandomQuote().getQuote());
    }
}

KanyeService

This service uses the MicroProfile REST client. This allows the application to map the REST call onto an interface structure, similar to Spring Data. You’ll notice that there is no reference to the HTTP target; that’s stored in an application.properties file; you don’t want to be embedding those kinds of values.

package au.com.gigliotti.quarkus;

import org.eclipse.microprofile.rest.client.annotation.ClientHeaderParam;
import org.eclipse.microprofile.rest.client.inject.RegisterRestClient;

import javax.ws.rs.Consumes;
import javax.ws.rs.GET;
import javax.ws.rs.Path;

@Path("/")
@RegisterRestClient
@Consumes("application/json")
public interface KanyeService {

    @GET
    @ClientHeaderParam(name="User-Agent", value="curl/7.54.0")
    KanyeQuote getRandomQuote();
}

Container Magic

The real super power for Quarkus is the building native images feature. All you need to do with this example is to run gradle buildNative to compile into a native executable. On MacOSX, Quarkus actually spins up a Docker container in the background to do the compilation, so you don’t need to configure GraalVM locally. The compilation on my 2015 MacBook Pro isn’t super fast, at approximately 2.5 minutes to compile, but this process is something that you’d only do when you’re ready to deploy into your container.

As a native executable, this application starts in 0.010 seconds, which is blisteringly quick.

Still In Beta

At the time of writing, Quarkus was at version 0.21.1, so it’s still very much in beta. I struggled to get the Gradle plugin working with Kotlin, so I needed to revert to using standard Java. Speaking of build tooling, the integration works nicely if you’re familiar with the Java ecosystem, but there is still a steep learning curve for newcomers. Saying that, the documentation is really top notch, as are the samples.

Conclusion

You probably want to wait to version 1 before running your production workloads using Quarkus. Saying that, it’s refreshing to see a viable alternative to Spring, particularly one with such a focus on developer productivity and cloud deployment options.

Testing with Gauge

Sat, 16 Mar 2019 17:11:00 +0000

End-to-end (e2e) testing is one of the most overlooked, underappreciated aspects of software delivery. A well written e2e suite gives you the confidence to develop and deploy quickly. A badly written e2e suite results in deliverables of unknown quality, leads to frustration, and inevitably project delays. As Michael Feathres wrote in his book, Working Effectively with Legacy Code:

To me, legacy code is simply code without tests. I’ve gotten some grief for this definition… I have no problem defining legacy code as code without tests. It is a good working definition, and it points to a solution.

The problem I have found with e2e testing suites, is that they are generally very difficult to setup and maintain. Particularly in the JavaScript space there are a bunch of existing frameworks and libraries available, but I feel that every time I try and do something I spend the majority of my time fumbling with different versions and configuration incompatibilities. It’s like trying to tape a bunch of cats together.

Enter Gauge, a new lightweight Open Source Testing Framework. It provides an opinionated CLI tool to allow you to rapidly develop tests, without having to worry about the underlying plumbing. Install gauge, select a language, start writing tests. A promise too good to be true?

Gauge Overview

There are a number of parts that makeup Gauge, and it is important to understand how each of them interact.

Specification Files

Gauge uses a Markdown based format for its specification files. It is similar to the Gherkin syntax used by Cucumber, but offers the flexibility of being able to have steps littered throughout more complex explanations. There is a fairly standard Scenario > Step structure, similar to Gherkin, and it supports familiar tagging and table syntaxes. Anything that you expect in Gherkin should be available in Gauge.

There are however, a couple of very powerful other features.

Concepts Steps: Allow you to combine re-usable logical groups of steps into a single unit. Think of having a “Login with user “bob” function, where you define the login as a Concept which you can reuse across different scenarios.
Special Parameters: It natively allows you to externalize large text In Files, or pass in large tables via CSV. No more enormous specification files.
VS Code Integration: The integration is second to none, with step auto-complete and the ability to handle execution from within VS Code.

Gauge Runner

Gauge itself is written in Go, but allows you to define your step definitions in a variety of other languages. There is official Gauge support for Java, C#, Ruby, JavaScript, Go and Python. The Java and JavaScript runners seem to be the most feature rich, and we will be looking at the JS runner because it supports Taiko.

Step Definitions

The step definitions provide the glue for the execution of your tests, similar to how they work for Cucumber. However, there are a couple of additional features that I would like to call out:

Filtering Hooks: Allow you to trigger filters based on tags associated with a scenario. So for example, you might setup a test differently if it is a production test instead of a local test.
Data Stores: Gauge provides a mechanism to store and retrieve data between step definitions. This is important, because Gauge allows your tests to be run in parallel.

Gauge CLI

The CLI is the heart of Gauge. It allows you to execute your specs as required, initialize your project, and interact with your test suite. The power is in its simplicity; you have one utility that you need to remember and that is it.

Taiko

Taiko is a Node library for interacting with Chrome, which on the surface sounds very similar to Selenium. Despite the similarities, its main difference is its smart selector system. The aim with this is that you treat the webpage itself as a black-box, and you interact with it like a user would.

Let us take a login scenario as an example, where a user needs to enter their username and password before proceeding. Using a traditional Selenium approach, you would use some kind of selector, generally a CSS selector, to identify the input boxes, and then write in the values as required. With Taiko, you state that you want to write a value into a field, and you select it by its onscreen label; this might even be a label above the actual element. Taiko then works out which element you require based on the action you are performing, and the data you have provided. You effectively are able to use the presentation of the page to interact with it, instead of having to worry about the underlying structure. If a developer inadvertently renames an element, or restructures some of the HTML, as long as the presentation is the same/similar, your tests will still function. I am sure there are scenarios where this won’t work effectively, so there are fallback options.

Another amazing feature of Taiko is the Interactive Recorder, with is a RPL interactive shell which allows you to interact with your browser dynamically. This allows you to verify the Taiko commands you would like to use, and will also generate the code for you when you are done.

Example

You will first need to install Gauge. There are plenty of package options available; for example on MacOS you can use brew. Once installed, all you need to do is run the following to initialise your Taiko based project.

gauge init js

And then running it is as simple as:

gauge run ./specs

You’re already cooking with gas. Now, say we want to have a test that goes to the Jhipster page and clicks the Sponsors link. You can use the Taiko RPL to generate the step definition for you. See below for an example of interacting with Taiko.

The code generated by the .code command can then be inserted into a step definition, or multiple step definitions. It is generally better to have generic step definitions, and then use detailed definitions for things that really require them. You can also always use Concepts to bundle up complex interactions into single steps, another useful feature of Gauge.

If we massage the above generated code into the following step definitions, we can make it generic.

const {
    openBrowser, write, closeBrowser, goto, press, text, contains
} = require('taiko');
const assert = require("assert");
const headless = process.env.headless_chrome.toLowerCase() === 'true';
beforeSuite(async () => {
    await openBrowser({
        headless: headless
    })
});

afterSuite(async () => {
    await closeBrowser();
});

step("Goto <url>", async (url) => {
    await goto(url);
});

step("Page contains <textToCheck>", async (textToCheck) => {
    await text(textToCheck);
})

step("Click link <linkText>", async (linkText) => {
    await click(link(linkText));
})

Then, the specification file can look like this:

# Fun with Gauge
## Go to JHipster
* Goto "https://jhipster.tech"
* Page contains "Jhipster news"
* Click link "SPONSORS"

Running it again via gauge run ./specs will have the test navigate to the website and interact with the defined elements. It also generates a pretty report for you so you can see what is up.

Final Thoughts

Gauge for me fills a gap in the market when it comes to fast, maintainable testing. It takes the lessons learned around Cucumber and Selenium, and removes the obvious pain points around them. It seems obvious that e2e testing would move in this direction, and I am excited to see where it is going to develop in the future. Go get Gauge, and start testing.

CKAD Exam Tips

Sun, 03 Mar 2019 00:00:08 -1000

I recently received my Certified Kubernetes Application Developer (CKAD) certification, and boy was it fun. I thought I’d put together some of my tips around my exam prep, as I’d only really dabbled in Kubernetes previously.

Learning Kubernetes Itself

Kubernetes in Action

To start with I decided to read through Kubernetes in Action by Marko Lukša. A tome at 624 pages, the book covers the basics needed for the exam, as well as additional sections regarding extending Kubernetes and multi-clusters. You’ll definitely want to run through the book at your computer, so you can punch the examples in and see how they react. It’s particularly important for topics such as the deployment handling. There are numerous Kubernetes release cycles a year, so some of the stuff in the book wasn’t entirely up to date. Saying that, Manning released a small update in February.

The book approach isn’t for everyone, and there are plenty of other introductory tutorials on the web, including a free introductory course from CNCF. Some of my colleagues who were studying at the same time preferred this.

Pick a Distribution to Use

Although Docker for Mac now includes Kubernetes, I think it’s better to use a more realistic distribution. I ended up using kops to deploy a working Kubernetes cluster on AWS. One of the neat features of kops is that it allows you to generate valid Terraform configuration files, which I then used to create and destroy the cluster as required. Kops can get expensive if you leave it running.

I focused on kops because I wanted to learn about how Kubernetes interacted with AWS resources, like EFS and IAM Roles.

Exam Specific Tips

Read the Curriculum Overview

Make sure you read the Curriculum Overview, as it defines the topics that will be covered in the exam, and the percentage of each topic on the exam. Make sure you understand each of the dot points, and how they are created, modified and examined on a Kubernetes cluster. All of them are relevant.

Handle your kubectl

You need to be proficient in using kubectl regarding the topics covered in the Curriculum Overview. In particular you need to know how to create things with a minimal amount of typing. Some resources need to be created via a YAML syntax, but for the majority of resources CLI arguments should be sufficient. In terms of CLI tips:

the `kubectl run` command is particularly powerful. Use it with the --dry-run -o yaml` arguments to output YAML files.
I recommend setting up a `k` alias for `kubectl` so you don’t need to keep on typing it.
The exam makes use of 3 clusters and numerous namespaces. Get comfortable in switching between clusters and namespaces with `kubectl config` otherwise you’ll waste a lot of time prefixing the namespace to your commands, and you will introduce the possibility for additional errors.

I spent a lot of time running through the exercises defined here, as I found it a great way to learn the commands. It doesn’t necessarily represent the exam question format, but I found it invaluable. You should be able to answer every question before taking the exam without hesitation.

Practice your Linux Foo

You need to be proficient in your Linux terminal skills. You’ll need to be able to pipe data around, ssh into different terminals, and be comfortable with sudo. Having a workable knowledge of vim or nano is important. You don’t want to be copying large amounts of text back and forward between the terminal and the exam notepad, so you need to be able to edit in place. There isn’t a huge amount of typing required, as you’ll be copying from kubernetes.io for references, or generating YAML from cli commands, but you certainly need to be able to quickly edit a file.

Bookmarks in Chrome

Have a good set of Bookmarks defined in Chrome, and have them available in the often unloved bookmark bar. You should ensure you have links for all the topics covered by the Curriculum Overview.

The Exam Itself

The exam itself is pretty time constrained. Take note of the percentage worth of each question, because the length of the question does not necessarily correlate to its scoring. I’ve spoken to some of my colleagues who suggested working from the last question backwards, but I found it just as effective starting from question 1, and then skipping over questions that I couldn’t quickly solve. You’ll want to get as many runs on the board so you can clear the required pass mark.

In addition, I believe the exam is automatically marked, so you need to be spot on with your spelling. Either copy names directly out of the question, or be very careful when you type them.

Conclusion

I highly recommend sitting the CKAD Exam, if only just for the experience. Hopefully some of these tips will help you in that quest.

Journey to ippon.acrossthe.world

Tue, 30 Oct 2018 00:00:08 -1000

The Ippon Australia team have been working hard over the last couple of weeks on a recruitment challenge, called Ippon Across The World. It’s a short game to test technical skills, without being too over the top, and hopefully it’s a little fun along the way. Give it a try if you haven’t seen it yet.

Err Why?

During the downtime between projects, it’s important to maintain team cohesion, as well as learn new skills. Self-paced learning is good to a point, but you learn so much more by having a tangible deliverable, and a target to work towards. We also have on-boarded some great new team members recently, and it was a great opportunity to get to know them a little better.

Design

We went with an 8-bit video game motif, because, well who doesn’t like 8-bit? We wanted to capture that original Carmen Sandiego vibe but with a Melbourne twist.

Architect

AWS is our default cloud provider, so we used a modern AWS design for our application. We wanted to leverage as many AWS services as possible, to give us a holistic view of how some of these newer components hang together.

CloudFront is incredibly important to hosting a modern application on AWS; saying that, it is very often ignored or misunderstood. It provides not just caching services, but also edge-content delivery and TLS support. You can even now run Lambda Edge functions as part of your content delivery strategy; we’re using it here for CORS header support.
Route53 is the DNS provider for AWS, both internally and externally. You can also purchase and manage domains through it. I particularly enjoy finding novelty domain names.
AWS Connect is a rather unusual AWS offering, and we wanted to try it out. It’s effectively a call-center in a box, allowing you to establish elaborate inbound and outbound call flows, including ML-based text-to-speech, as well as human operator steps.
S3 static file storage for the website and Java libraries.
API Gateway for rate limiting and routing requests to Lambda.
Lambda for serverless API processing. The Lambda functions are primarily used to respond to answers and provide the next URL to follow, as well as for processing analytics events.
DynamoDB is used for storing temporary data, which we’re cleaning up with the automated TTL function. For those that are unfamiliar with DynamoDB, it’s a scaled NoSQL offering from AWS.
Pinpoint (and Cognito) used for analytics; tracking primarily website interactions.

User Interface

We decided to use raw JavaScript with GSAP for the animation components, with a focus on modern desktop browsers. No legacy Internet Explorer support here. Apart from the animation aspects, we also used Web Crypto API for one of the challenges, which was pretty exciting to see in action.

CI/CD Pipeline

No project is too small for a CI/CD pipeline. It makes life easier if you’re deploying something more than once, and it’s important when you have a number of people wanting to make a lot of little changes quickly.

We used Terraform for setting up the base infrastructure, and then a combination of shell scripts for updating deployed assets. Gitlab, which is our source-control tool, has an amazing pipeline mechanism for defining build processes, and we use that to push changes into the various environments when a single commit occurs.

What’s Next

Shipping an application is not the end of the journey, it’s the beginning. We’re collecting usage metrics via Pinpoint, which allows us to see how far users (anonymously) get through the game. We will be offering slight variations to the challenges depending on skillsets, and potentially offering a harder difficult level for the clever cookies.

We certainly enjoyed putting this together and we hope that you enjoy playing it.

Originally published https://www.linkedin.com/pulse/journey-ipponacrosstheworld-gerard-gigliotti/.

Open Banking, Open Opportunities

Mon, 23 Jul 2018 14:01:02 +0000

Open Banking represents a real paradigm shift in the ownership and use of customer data. It not only provides significant opportunities for consumers, fintechs and the banks but also represents a precursor to broader data ownership changes beyond just the realm of financial services.

Scott Farrell defines open banking as giving “customers a right to direct that the information they already share with their bank be safely shared with others they trust.” Traditionally viewed as an asset of the organisation, empowering consumers to be able to easily, and safely, share their data sounds like a really great idea. However what are the likely consumer implications?

If significant, democratising changes of the past are anything to go by then consumer uptake will be slow. Indeed, just take a look at the “choice of fund” legislation in superannuation which is almost 15 years old and still sees relatively few workers chose anything but their employers default fund. Another recent example would be the latest advancement in the Australian banking landscape, the NPP, allowing almost real time payments is almost unheard of by consumers. Some of Australia’s largest banks haven’t been able to integrate it into their offerings as yet.

As such, short term adoption of Open Banking is likely to be limited but it does put in place a foundation with the potential to deliver significant consumer value well into the future. To appreciate the value of the proposition it’s worth considering the examples of outcomes it could enable for customers.

Customer Value Add Services

One of the more interesting points in the Open Banking report was Recommendation 3.3, which states that “… data that results from material enhancement by the application of insights, analysis or transformation by the data holder should not be included in the scope of Open Banking”. What that means is any value add or proprietary processing you add as a bank to a customer’s transaction data does not need to be shared. The opportunity present, therefore, is being able to provide additional services on top of the underlying data in unique and compelling ways.

For example, imagine as a bank being able to provide proper categorisation of transactions to help drive financial outcomes for a customer. You could tell a customer they spent $24 on coffee this week, which was $4 over their budget. Or you could provide a kind of financial health check service to ensure they’re using the most optimal product for their situation. Providing value add services increases customer loyalty, and discourages Fintechs from plugging obvious gaps in your offering.

Data-Driven Acquisitions

A less obvious benefit of a free-flowing data approach, is the ability to use data for acquisition purposes. Imagine that you have a credit card with Bank A, and you’re applying for a new credit card with Bank B. By leveraging the Open Banking platform, Bank B will be able to incorporate the financial transaction data from Bank A as part of its signup process, and leverage that data to provide concrete examples of how their product is superior. For example, Bank B could show that “transaction X” would have resulted in “Y” number of reward points. The transaction data could even be used to target the kind of product being applied for, say a “Platinum Card” instead of a “Gold” card. A more advanced version of this approach could leverage the data supplied to customise the product specifically for a customer.

The key with this approach is to demonstrate the value that you can deliver for your potential customer immediately. A flashy brochure-wear website, showing picture after picture of near identical credit cards is fine, but that doesn’t differentiate your brand; that’s a transactional cost of doing business. Being able to effectively categorise data rapidly, using either fixed or machine-generated rules, could be used to significantly speed up the signup process, reducing the number of dropouts and increasing conversions. Looking at one of the big four’s credit card signup forms, they have 6 sections relating to finances, including a 100 word description of what an expense is; this could be massively simplified with Open Banking data.

Fintech & Developer Friendly

There is serious opportunity in being the bank that embraces the Fintech and Developer community in a meaningful and ongoing way, through both brand exposure, as well as from new customer streams. What this means is investing in developer advocates, relevant API documentation, sample code, and support channels for developers. If developers find it easier to build products using your services and data, and you make the experience compelling for them, they’ll be more likely to continue to use your products, and potentially branch out into using your products in other areas.

Open Banking will dictate a minimum set of standards that will need to be adhered to, but that shouldn’t be the target for any bank. Providing additional data points, or even write based access for some services would be a huge carrot for Fintechs. Imagine for example, a Fintech company which helps people manage and track their savings goals. If your bank allowed the Fintech company to transfer money between a customer’s various accounts, they could leverage and recommend your bank’s services to its customers. A symbiotic relationship in this case would benefit all parties.

Conclusion

The premise of Open Banking is consumer empowerment, industry invigoration and driving competition in the banking sector. As such, it is somewhat ironic that, at least in the short term, the biggest beneficiaries of this approach are likely to be Australia’s big four banks.

There is no doubt that they will be burdened with the cost of implementing approaches that facilitate the operation of Open Banking but, in the process, will also empower themselves to take advantage of this in ways that were not historically leveraged. This is a massive opportunity to drive significant engagement with their existing client base and thus reduce churn.

Further, should some innovative fintech startups create value in ways unforeseen by the Australian big four, their sheer market power and financial strength means that they could most probably play the fast follower by replicating the proposition or just acquire the company. Again, let’s consider the example of peer to peer lending, another approach to “opening up” the banking sector, where those platforms that were successful got bought, or invested in, by the big banks.

As an initiative Open Banking should absolutely be applauded. It creates a foundation that should drive positive changes in banking over the short, medium and long term. The ultimate success of the initiative should be measured by consumer adoption and the flow on benefits that ensue in opening up consumer data in markets beyond just banking.

AWS AppSync is coming for your API

Sun, 17 Jun 2018 00:00:08 -1000

According to the introductory post,

AWS AppSync is a “serverless GraphQL service for real-time data queries, synchronization, communications and offline programming features.”.

I know that statement doesn’t feature every 2018 buzz word, but it is getting pretty close. But what does it actually do?

If you’d like to read more about AWS AppSync, head over to the Ippon Blog. If reading isn’t for you, the associated pony-heavy sample is on Github here:

It's about the little (internet of) things

Sun, 14 Jan 2018 00:00:08 -1000

After moving into my new place I decided that I wanted to start down the treacherous path of home automation. I’d dabbled with a WeMo switch previously (see here), but that was more of a toe in the water. Now I wanted to jump in. My hesitation with home automtion/IoT in the past has been that it’s difficult to find a use case where it would actually enhanced my life. In addition, I don’t want any IoT automation to interfere with the day-to-day operation of how a reasonable person expects the device to operate. For example, if you’ve installed Philips Hue lights throughout your house, and your automations don’t work because somebody has turned off your globe at the switch, that’s a problem with the automation, not with the switch.

I luckily live near a tram line, and I catch the tram to get to work. Melbourne’s tram network is run by Yarra Trams, and they produce a pretty good app which provides arrival information for tram stops. Just an aside, the arrival information isn’t calculated by GPS, but rather a series of radio-based timing gates relative to your stop. The problem I have is that even during peak frequency, the trams are spaced 10–12 minutes apart. You don’t want to have to wait at the stop for 10–12 minutes. As a result, when I get ready in the morning, I usually have the iOS app up checking the tram arrival time. Constantly sliding down to refresh is giving me RSI, and although there is a notification system, it requires my input initially. This is my use case!

Now I can remember Werner Vogels at last year’s AWS Architect & Developer Day in Melbourne talking about natural interfaces, and this is a perfect example of where one can apply. My idea was to use a connected light globe, in this case one from LIFX (more on that later), and connect it to the Yarra Trams API. As the tram got closer, I wanted the light to get brighter, and then when it was absolutely time to leave, I wanted it to flash to warn me.

Parts

Yarra Trams API

Yarra Trams had a public API before it was cool, so although it’s SOAP-based, it provides data for all your tram network needs. Bang in the stop number and you’re good to get your predicted arrival information: GetNextPredictedRoutesCollection . Before you call that, you need to get a ClientGuid by calling GetNewClientGuid, and then you’re good to go. Simple.

LIFX

One of my fundamental home automation rules, is that the devices must be controllable locally. I don’t want to have to communicate over the internet to talk to a device which is running on my local network; it’s unnecessary. LIFX have a pretty robust looking Remote API, but luckily they also have a local LAN protocol for communicating with globes. LIFX Globes are standalone, so the protocol has a discovery mechanism using UDP for finding globes. NPM has (almost) everything, including a NodeJS implementation of the protocol.

Docker

Although Docker is supported on Raspberry Pesis , Docker Machine doesn’t just work out the box. I followed this guide to trick it into working. Probably not production grade, but I haven’t had too many issues with it.

Lazily Gluing It Together

My implementation is fairly rudimentary; it’s on Github. The application runs on a timer, and effectively runs for 30 minutes over my departure window. When a tram is within 7.5 minutes, the light is turned on, and then as the tram gets closer, the brightness increases. When it’s really close the light pulses to signify it’s now time to leave. If the tram is arriving in less than 4 minutes, the light is then turned off, as I won’t be able to make it in time.

Conclusion

The real benefit of this “thing” is that it allows me to continue my morning routine without needing to constantly refresh my phone for tram arrival information. It’s non-invasive, yet still informative. This is how we should think of IoT devices, they need to enhance, not inhibit our lives.

Putting the FUN into Functioning CI/CD

Sun, 25 Jun 2017 00:00:08 -1000

This is a presentation I gave at the inaugural Melbourne JHipster meetup. A big thanks to all you came along to hear me ramble for 15 minutes or so.

The itsy bitsy source code is located here.

Intro to JHipster

Wed, 03 May 2017 00:00:08 -1000

My first look at JHipster

So I’ve just recently switched jobs, and one of the most interesting aspects of my new employer, Ippon, is that they develop JHipster. One of my first tasks was to familiarise myself with the framework, what it is, how to use it, and how to deploy it, and boy was I in for an adventure. On first glance it looks like a standard Yeoman based scaffolder for Java applications. You answer some questions in the CLI, you hit enter, and boom, out comes a fully formed Spring-Boot Java application. However the more I looked into it, the more I started to enjoy all the included goodies.

Features

Microservices

Instead of just generating Java Applications, JHipster has an entire microservice approach for deploying applications. People talk about microservices as a nebulous cure to all technological woes, but the reality can be quite complex and difficult if not properly managed. JHipster presents a vision of how to build and deploy a microservices solution, based on Spring Cloud and Netflix OSS. It might not align to everybody’s vision, but it’s a solid, production-ready foundation. In its most basic configuration, there are three main components to the JHipster microservice stack:

Gateway: The naming of this component confuddled me initially, but it’s effectively an API Gateway Server and a static file-server for your SPA.
Microservices: The Java based microservice heart. Can be deployed in a variety of configurations, including databases and the like.
Service Discovery: Method for services to discover one another, and share configuration details. There was initially a JHipster developed JHipster Registry, but Consul can now be used in place of that.

Note: There is also the option for an authorisation server, but that’s a discussion for another time.

Entity Creation

JHipster has an inbuilt UML like language which allows you to model relationships between entities. I’ve struggled with these kinds of tools in the past (I’m looking at you Rational Software Architect), but this one is pretty intuitive. It generates helpful boiler code not just at the entity level, but also full CRUD level endpoints. It also generates sensible Java code which won’t make you vomit. There is an online tool called JDL Studio, which allows you to model your entities. You then then feed the generated file into the entity sub generator, and out comes all the entities beans, Liquibase scripts, endpoints and tests you could ever dream of.

Docker

Docker is used for spinning up dependencies, and can be used to package your application. Docker make everything better *exclusions apply.

Testing

Out of the box it contains options for a bunch of testing frameworks, on both the Java and JavaScript side of the fence. Tools like Protractor are available out of the box with PhantomJS. I’d probably swap that out for a Chrome based Docker image with a Remote Web Driver until headless Chrome is available.

CI Generation

Need to add a pipeline to Gitlab? Done.

Deployment

Out of the box it’s trivial to run a JHipster application through docker-compose. Everything is there with a nice little bow on it. If you want to run something in production, there are a bunch of deployment options in that space. Due to my pathological love of containers, I gave the Rancher option a go. Due to its need to have access to a Docker Registry, it’s not quite as straightforward as some of the other interactions, but it does work as it says on the box. I ended up using Amazon’s Container Registry, which worked after following this.

Where to use JHipster?

The obvious starting place is the microservice generator. It’s a quick and relatively painless way of generating Java microservices. I’ve worked for a number of organisations which have developed similar, but not nearly as feature rich tools internally, and it’s a great way of reducing all that Sprint 0 project setup malarchy. I mean who doesn’t like wiring up Spring Contexts?

The gateway component is an interesting concept, but certainly for the customers I’ve worked with, a harder sell. They’ve spent a number of years investing in monolithic API gateways, and there would certainly be a reluctance to move away from that towards a more feature-centric API and application approach. Saying that, the ridginessness of the monolithic API gateway and the constraints it can put on innovating front end applications may foster a willingness to push gateway functionality back into application teams. This would be where the JHipster Gateway would really shine, particularly for JVM centric organisations.

AWS Lambda, How to Tape a Bunch of Cats Together

Thu, 06 Oct 2016 00:00:08 -1000

I was lucky enough to attend an AWS Architect and Developer Day in Melbourne recently, and as you’d expect, Lambda was an underlying theme. I didn’t attend a single presentation in the developer track that wasn’t plugging Lambda in one form or another. Lambda for micro-services, Lambda for Alexa, Lambda as a database event trigger, Lambda for transactions, Lambda for Lambda. Lambda. Lambda. λ.

I ❤️ Lambda. I think it’s an excellent tool in the ever increasing bag of tricks that is AWS, but I’m a bit hesitant when Amazon talks about it as the golden hammer which is going to fix all your problems. For starters, you need some rigor around how you develop and deploy your functions, particularly in large development teams. For example, the multi-stage functionality offered by the API Gateway are great, but no enterprise customer is going to allow you to run a “DEV, TEST and PROD” environments in a single AWS account. Never going to happen. Ever. And nor should it. Separating environments (or stages) across multiple AWS accounts, allows for separation of responsibilities that large organisations love, and are generally required to have. So how should we look at using Lambda in a large company?

Developers get their own AWS credentials, in a Linked Account

To use Lambda, Developers need AWS credentials to deploy functions, hook in event triggers and explore the various managed-services that AWS has to offer. If your company is using AWS, and each one of your backend developers does not have access to an AWS account, you are doing it wrong. That’s not to say that it’s just a developer free-for all, where every man and his dog can spin up new P2 instances on a whim, but there needs to be sensible controls, and monitoring in-place to strike a balance. I’d recommend a set of IAM permissions to limit accounts initially to a subset of managed services, with an appropriate tagging system in place to identify resource ownership, and a fairly aggressive monitoring strategy to ensure costs don’t blow out, and developers aren’t spinning up bot-nets.

Python & NodeJS Only for API-Gateway Events

I’ve mentioned this in a previous post, but the performance of Java in Lambda is a non-starter if you’re wanting to use it behind the API-Gateway. Until AWS improves the performance of the cold-start mechanism for Java based Lambda functions (see here), it’s unworkably slow. You can happily use Java for non-time sensitive operations, such a Dynamo DB triggers, but just don’t use it for API Gateway functions. If you’re thinking about writing another function to keep the container warm by triggering it every 5 minutes, back away from the keyboard, go get yourself a coffee, come back, and then spend that time you were going to use writing the trigger function to rewrite your code in NodeJS or Python. You’ll be much happier, everybody will be much happier.

Lambda Function Migration

The following diagram demonstrates an approach for migrating Lambda functions between environments. There could be multiple environments in a single AWS Account, such as with the Test AWS Account, or there could be a single environment, just like production. Why the multiple account approach? It separates concerns, and ensures that breaches or problems in lower accounts have no chance of affecting production. You don’t want to be in the scenario where somebody accidentally deploys a development version of a Lambda function into your production API Gateway stage, something that can easily occur if you’re using a single AWS Account with poorly configured IAM permissions. In this model, there are multiple stage-gates to ensure only verified changes make it to production.

CI/CD can sometimes be bandied about as a single term, but it’s two distinct practices, which you may wish to use two different tools to achieve. For the CD component A lot of enterprise customers will just use a bunch of Jenkins servers together, which can work, but generally adds a lot of maintenance overhead. Other tools, like Spinnaker or AWS CodePipeline are generally a better fit. The above diagram has the pipeline tool being powered off an artefact deploy to S3, which is then deployed out to the various accounts and environments.

Lambda for MVP

The real reason to use Lambda, is that it’s fast to develop & deploy. You can quickly write a small set of functions to prove out a hypothesis , and then deploy them to run in a production-ready configuration. You’re not worrying about infrastructure, you’re not worrying about scaling, you’re focusing on you business problem at hand. You can code, deploy and analyse faster than you’ve ever been able to before, which makes it perfect for Minimum Viable Products. Every company should is interested in the flexibility.

If you’re Lambda function is being executed too many times a month, and it’s no longer economical to run in the Lambda, then that’s a great problem to have, as your product is getting used. You can then start to look at moving off Lambda, onto more flexible infrastructure. I’ve worked on projects which sit on thousands of dollars worth of infrastructure a month, but are only called a few thousand times a day, which is just not economical. You won’t be running your internet banking infrastructure on Lambda, but you might want to run your fraud complaints system.

Conclusion

As I said, I ❤️ Lambda. I jump at any opportunity I get to use it. But large companies that are still skeptical of its magical powers are missing out on the opportunities to innovate just as quickly as that startup around the corner. It doesn’t need to be as footloose and fancy-free as Amazon makes it out to be, and you can still use it in a controlled and sensible way. If you’re an IT Architect, and you’re reading this, come to the dark side. The water is lovely.

The Enterprise JavaScript Conundrum

Tue, 26 Apr 2016 00:00:08 -1000

I’m a full stack developer in the way that McDonalds is a breakfast restaurant; yes, they serve breakfast, but only really from 5am to 10:30am. I only get into the nitty gritty of the frontend when I absolutely have to, and it’s certainly not a place that I generally spend a lot of time. Saying that, it’s important to keep up with what’s out there, at least from an awareness perspective. The frontend darlings at the moment are clearly Angular (particularly in the Melbourne market), and React. I’ve worked with Angular (1.3) previously, but thought it was time to see what React was about, as there’s a lot of talk about it around the office water cooler. It became a rather interesting adventure down the rabbit hole…

First up, the tooling around JavaScript is changing at a neck-snapping speed. If we look at Babel’s installation instructions (more on that later), there are 30 configuration options. Kudos to the Babel team for supporting such an impressive plethora of tools, but it makes selecting an “optimal” configuration challenging. Then there are the tools themselves. Tools such as Grunt which were du jour a couple of years ago have been relegated to almost a pariah status amongst some of the new libraries. If you’re not starting with Webpack (maybe with a hint of Gulp), then you’re not starting on the right foot. Webpack itself has a fairly steep learning curve, but once you’ve got it setup correctly you wonder why you ever lived without it. The automatic refresh capabilities with the inbuilt dev-server are super helpful. Saying that, it doesn’t “just work” out of the box, and you still need to tinker around with it before it comes to life.

To use the latest version of JavaScript, ES2015(ES6), you really need to use a transpiler like Babel to convert the code down into something more universally implemented across the various browsers + node. If for example you want to use the new arrow function it doesn’t work on Mobile Safari, which is not going to get you many friends on mobile.

It’s a pretty exciting time to be a JavaScript developer, but I’m concerned that the pace of innovation isn’t compatible with the approach taken by large enterprise customers when it comes to maintaining their websites/applications. Companies in Australia might be shifting their development practices to being more agile-oriented, but their approach to projects and code maintenance is still very much stuck in the waterfall. You might work for n sprints to deliver a new application, where you’ve used something like Webpack, and React, but once it’s delivered it’s then handed off to a Business As Usual (BAU) team which doesn’t necessarily have the funding to continually update and maintain the codebase. I’ve worked on applications during the last year that will be in one form or another still being used by their respective clients for the next five years. What happens if they need to add some functionality in five years time? Will they still be able to download and use the Webpack tooling with the required dependencies?

The solution is understandably to change the way codebases are developed, maintained and funded within companies, but it’s hard to encourage these practices without some kind of “crises” to facilitate the discussion. I think it’s about curtailing over excited developers to a certain set of approved tools, and then making sure new versions or approaches are rolled out across every living application. I was working with a client last year where they were trying to run multiple, older versions of Angular applications on the same page. These problems are a development nightmare and ultimately end up in sub-standard user experiences. A trickle upgrade approach needs to be mandated for all enterprise projects, especially when dealing with consumer facing JavaScript projects. It might not be as exciting now, but you’ll be thankful when you’re getting called up in five years to update your business critical Angular 1.5 app.

AWS Lambda — Bringing Java Back

Mon, 24 Aug 2015 00:00:08 -1000

So I spend the day toiling away at the big-bank enterprise Java coal face, so I’m generally not very motivated to play with it when I get home. I’ll sheepishly admit that I hadn’t even installed Java 8 onto my laptop until recently. It’s not that I wasn’t excited about Java’s lambda support, the streams processing, or Nashhorn, it’s just that it’ll be years before I’ll get to actually use them for a corporate customer.

I was pretty excited by the AWS Lambda announcement back in June announcing Java support. So I thought I’d give it a try. After playing around with it, for a bit, I’ve written up a list of hints and lessons from my first week playing with it:

Make your zip file small….

Every time you make an update to your Lambda function, you need to update the entire associated JAR; no in-window text editing for you. If you’re on a terrible ADSL connection, like I am, then failing to do this will result in a lot of heartache. If you’re relying on other AWS services, ensure you pull in only the AWS dependencies for the services you require. Do not use the standard catch all AWS SDK For Java. It comes in at over 45 megabytes, and it’ll make you cry. Amazon is kind enough to split each service into its own little package. Also use S3 to store your binaries; the sync appears to be much faster than the general Lambda upload.

Warm up

Lambda functions take a while to warm up. If you haven’t used it for a while, or you’ve just deployed it, be prepared to see a long response time on your first request. This is fine for when you’re just playing around with it from the AWS-console, but it can be a little annoying if you’re invoking it from the API Gateway, which has a fixed 10 second timeout. 504s abound. For production, I imagine you could use some kind of heartbeat for the Lambda function, at least to have a minimum number of instances running at all times.

Context of the Request

One of AWS’ strengths is the robust identity and access management framework. When your Lambda function executes, it automagically has the access granted in the associated roles and policies. So for example, if your function is trying to insert data into a DynamoDB database, a policy like this will enable you to do so.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1428341300017",
            "Action": [
                "dynamodb:DeleteItem",
                "dynamodb:GetItem",
                "dynamodb:PutItem",
                "dynamodb:Query",
                "dynamodb:Scan",
                "dynamodb:UpdateItem",
                "dynamodb:ListTables"
            ],
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Sid": "",
            "Resource": "*",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Effect": "Allow"
        }
    ]
}

The other side of the coin is making sure that you make use of those automatic credentials. I wasted hours, hours, trying to make my Lambda function connect to DynamoDB. My mistake? Not using the default constructor to access the credentials. I’m sure the seasoned AWS developers will be able to tell you exactly which AWSCredentials implementation to use, but for young players the default constructor is your best friend.

Cloud Watch — Console Debugging

Lambda can send all your logging statements to Cloud Watch. There are really two ways to log. You can use System.out, which will make seasoned Java developers turn over in their respective graves, or you could use the “context.getLogger()” method available in the context. Of course, the issue of having to pass your logging mechanism around the stack isn’t ideal, but if that’s the pattern, that’s the pattern.

Forget the 128 mb option

The base 128mb option is great if you want to write a hello world example, but if you’re doing anything else other than that, you’ll want the 256mb memory allocation as a minimum.

AWS Lambda really turns the Java application deployment model on its head. It abstracts away the complexities of deploying production scalable apps, and wraps it up with a nice little bow. It might not be replacing your legacy stack anytime soon, but it’s certainly a great glimpse into the future.

Enterprise IT, be careful where you tread

Thu, 29 Jan 2015 00:00:08 -1000

The client I’m working with at the moment is going through a re-organisation of its delivery teams, and as part of the process they’re also re-evaluating the tooling used for projects. The management has given the group free reign in dictating the frameworks and tools we want to use moving forward, which is liberating, but also potentially fraught with danger. As technologists it’s so easy to jump straight into the “shiny shiny” (thanks Paul), and ignore the reality of working within a risk adverse enterprise IT space. Things to consider:

Maintainability & Supportability: Software in large organisations is used for a very, very long time. The project I’m working on at the moment is replacing a web application, which is over 14 years old. It’s not necessarily the prettiest thing, but it’s very stable, and more importantly it’s maintainable. People with the skillset are easy to find, and the platform is still supported, which leads to the next point.
Only proven technologies need apply: I love playing with the latest and greatest at home, but if it hasn’t been proven to work somewhere else, I’m not interested. For example, we’re building a new web-app, and I’d love to write it in Golang, but if I can’t get a Microsoft supported database driver, it’s a hard sell.
Team Velocity: Pick technologies that your team knows well. If you’ve got a bunch of Ruby developers, build your solution in Ruby. Java developers, a Java solution. This isn’t necessarily exciting, but it ensures you’re getting results sooner.

It’s fun to be able to branch out and learn new tools, but the above should always be considered when selecting new tooling for a project. Is it still going to be fun to work with in a decade’s time?

Adobe Experience Manager & Docker — Part 2

Mon, 01 Dec 2014 00:00:08 -1000

So last time I left you, I’d put together a post on getting an Adobe Experience Manager 6 Author instance running in a Docker container. It was a great way for me to get my head around Docker, while also playing with the technology stack I’m currently using at work. Getting it running was fun, but by itself a single Author instance is a little pointless; you can achieve the same thing by clicking on the AEM JAR.

What isn’t pointless, is an Author+Publisher+Dispatcher stack, all running in individual Docker containers. To orchestrate all this madness, I’ve put together a Fig YAML file within a GitHub repo to co-ordinate the various containers. Fig basically is an orchestration utility for Docker to help describe relationships between containers.

Why Docker? It allows you to have a portable full AEM stack that you can distribute amongst your development team. The power of containerising the various AEM components allows you to ensure a consistent and repeatable configuration between team members. I definitely wouldn’t be using this in a production setting, but it’s perfectly fine for development.

Getting Started

Make sure you’ve got Docker, Fig and boot2docker available; boot2docker is only required on MacOS X or Windows.
Clone/Download the Github repo.
Copy your the installation media and license files into the author and publisher directories. The install media must be given specific names depending on location. cq-author-4502.jar for author, and cq-publish-4503.jar for the publisher.
Navigate to the basedirectory, and run: fig up.
Go get a coffee.
If all has gone to plan, the stack should be running. If you’re using boot2docker, you’ll need to reference it via its IP ( you can find it using boot2docker ip ). The ports are as follows:

Dispatcher: Accessible over port 80 or 443. For example: http://192.168.59.103/content/geometrixx-outdoors/en.html .
Publisher: Accessible over port 4503.
Author: Accessible over port 4502.

Hopefully you’ll be able to enjoy your full stack AEM development.

A couple of notes/caveats

Link relationships in Docker are currently only one way. There are a couple of ways around this, but for the moment, I’ve set the build to only allow one way communication between containers. Ie, author->publisher, dispatcher->publisher. The Fig DNS stuff looks interesting.
The standard boot2docker image does not have enough disk or RAM to get this working correctly. Look at re-initialising it with at least 40gb of diskspace, and at least 4GB of memory: boot2docker init -m 4092 -s 40000 .
I downgraded the base image from Centos7 to Centos6. I was having trouble installing Apache in Centos7, but it seemed to work fine under 6.
The author->publisher replication configuration seems to be lost after setup. If it can’t connect, you’ll want to check that “publisher:4503” is set in the replication settings.

Adobe Experience Manager & Docker

Mon, 08 Sep 2014 00:00:08 -1000

Adobe Experience Manager (AEM) is the current darling of content management systems for large enterprise clients. The software’s relatively slick interface, and Adobe’s relentless sales team mean that you can’t swing a cat in Melbourne without hitting a company looking at deploying AEM.

I’ve been working on an AEM project the last view months, but to learn more about AEM deployment process, I decided that I’d try and get it running within a Docker container. Getting a basic author instance running on your workstation is as easy as double clicking the installation file, but more complex deployments with publishers and dispatchers are more challenging. However, to start with I’ve put together a basic Docker AEM author image as a proof of concept.

Automating the AEM installation was challenging enough, as I ended up needing to use a Python script to monitor when the installation was actually finished; something that Adobe seems to hide in a command line argument and a HTTP callback. What happened to the days when an installer could quit all by itself?

My complete inexperience with Docker also introduced some infuriating hindrances in the process. I got all excited by the plethora of recipes on the Docker registry but it took me a while to discover that not all images are created equal, and that not all images are compatible. I was quit pleased when I saw I could get a Java image (“dockerfile / java”), but I wasn’t too pleased two hours later when I finally noticed that it was based on Ubuntu, and therefore wasn’t accessible in my Centos image. A valuable lesson indeed.

I’ve uploaded my Dockerfile to the Docker registry, for your enjoyment. Hopefully it’ll give you a hand with your AEM projects.

Fitbit Powered Home Automation

Tue, 18 Feb 2014 00:00:08 -1000

My new employer, Odecee, runs innovation events every quarter where employees are given a hectic 48 hours to work on an idea. This time around I worked on a project leveraging Apache Storm, which was great fun, but the thing that really caught my attention was the white-paper research being done by the iBeacon team. In particular, the fact that the Bluetooth LTE software they were using on their iPhones was able to detect my Fitbit Flex, through the office floor. A little creepy that my location could be tracked by anyone with an iPhone in the vicinity, but it got me thinking, could I use this data leakage for good?

Home automation was the first thing that came to mind. Electricity prices in Australia are outrageously expensive, and I was looking at a way of controlling my appliance usage using a WeMO Switch. Traditionally you interact with these switches using either the WeMO Switch + Motion product to detect movement, by manually controlling the devices with your phone, or more recently using IFTTT’s location services. All these approaches are cumbersome. I want my house to know when I’m home, and this is where the Fitbit comes in. I wear my Fitbit almost all the time, so it’s a perfect way of detecting my presence.

So the plan was to use a Raspberry PI with a Bluetooth 4.0 USB Dongle to scan for my Fitbit. I would then connect the Raspberry Pi to IFTTT (somehow), which has a handy WeMO channel for controlling the plug.

I wrote a tiny Python script which scans for the presence of my Fitbit using the Bluez hcitool utility. When my Fitbit’s MAC address is detected, it records a timestamp against the reading. Then, every 10 seconds it checks the device and timestamp map. If a device hasn’t been detected for the last 30 minutes, it sends an email to IFTTT telling it to turn the WeMO switch off. If a device has been detected, and the script previously recorded an off state, then it sends an email to IFTTT to turn the WeMO on.

Why email? Well although IFTTT allows you to connect all manner of web-sites together, there is no straightforward way of adding your own data in, particularly via some kind of API. To get around this, email seemed to me like the easiest option. So my two IFTTT plans wait for emails, and when an email is received with a #deviceOn/#deviceOff tag, the WeMo will be toggled on and off.

There you have it, the basis for a Fitbit powered home-automation system. It seems to be working relatively well at the moment, and I haven’t seen any noticeable drop in my Fitbit battery. Hopefully there’s a drop in my power bill.

Meow you doing

Mon, 25 Feb 2013 00:00:08 -1000

So for my birthday last year, I was pleasantly surprised to receive a Lucky Cat, traditionally refered to as a Maneki-nek in Japanese. You typically see them in businesses, beckoning you to come inside. The cat, powered by two AA batteries, slowly waves its hand at you. That is, until it runs out of batteries.

Annoyed at having to change the batteries, even with rechargeables, I ventured a little outside my comfort zone, and splashed out on an Arduino Uno which can be wired up to operate various electronics. It’s a bridge between the hardware and the software worlds. I like a good bridge.

Arduino & Sketch

Knowing almost nothing about circuitry, I hacked in a design that came within the Sparkfun Inventors Kit documentation. After spending ages tinkering with the wiring, and a bit of Macgyver style blue tacking, I was able to get the Arduino to power the beckoning hand.

The Arduino takes has its own special language, which you use to write “sketches”, which are then uploaded to the device. My sketch, which is rather primitive, is as follows:

/*
  Cat.sketch
Sketch is used to read in an integer value from a serial connection,
  and then enable PIN 9 for the supplied amount of time.
*/

// Port 9 is being used to control the cat
const int CAT_PIN_OUT = 9;
/*
  Sketch initialisation.
*/
void setup()
{
  pinMode(CAT_PIN_OUT, OUTPUT);
  //Sets up serial connection
  Serial.begin(9600);
}

/*

*/
void loop()
{
  //The duration, in milliseconds to elave the cat arm running for
  int timeout = 0;

/*
    Puts the code into a never ending loop
  */
  while(true)  // "true" is always true, so this will loop forever.
  {
      // First we check to see if incoming data is available:
      while (Serial.available() > 0)
      {
        timeout = 0;
//Read in the timeout value from the serial input.
        timeout =  Serial.parseInt();
        Serial.print("Timeout ");
        Serial.print(timeout);
        Serial.println();
        //Turn the power on, wait the "timeout" of time, and then turn it off
        if(timeout > 0)
        {
          analogWrite(CAT_PIN_OUT, 255);  // turns arm On. 255 means full power
          delay(timeout);   //leaves cat arm running for as long as the timeout  millisecconds          analogWrite(CAT_PIN_OUT, 0);  // turns arm off
        }
}
//Short delay before trying to re-read the serial connection
      delay(50);
  }
}

So this code basically just reads in an integer value from the serial connection, and then turns the cat’s arm on for the supplied amount of time. The next step is to invoke the code from the computer.

Python & Twitter

Communication from your computer (in this case, my mac), to the Arduino is relatively simple. All you need is a library called pySerial (http://pyserial.sourceforge.net/) and you’re ready to go. So I grabbed the library, and I enjoyed stopping and starting my cat, but I wanted a little more from it. And that ‘more’ manifested itself in the twitter streaming API(https://dev.twitter.com/docs/streaming-apis). The streaming API offers a tiny, miniscule sample of twitter;’s global stream of tweets, which you can filter on. So after a bit of searching I stumbled across tweepy (https://github.com/tweepy/tweepy), a python API that has access to the streaming API. Without further ado:

import serial
import time
import sys
import inspect
from Queue import Queue
from threading import Thread
import time
import tweepy
from tweepy.streaming import StreamListener
from tweepy import OAuthHandler
from tweepy import Stream
# Twitter consumer and access keys
consumer_key="<INSERT_HERE>"
consumer_secret="<INSERT_HERE>"
access_key = "<INSERT_HERE>"
access_secret = "<INSERT_HERE>"
# Wave duration
WAVE_DURATION = '3000' # in milliseconds
SLEEP_TIME = 3 # in seconds
#List of twitter terms
STREAM_FILTER='#cat,#cats'
# Establish authentication information
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_key, access_secret)
api = tweepy.API(auth)
# serial connection to the arduino
s = serial.Serial(port='/dev/tty.usbmodem1d21', baudrate=9600,timeout=4)
# Queue for work thread. We only want the thread handling one "wave"
# item at a time.
q = Queue(maxsize=1)
# Thread which handles the cat's arm waving
def worker():
    # Loop forever. Retrieve item from queue, and then sleep
 while True:
     item = q.get()
      time.sleep(SLEEP_TIME)
      s.write(WAVE_DURATION)
      s.write('')
#starting cat thread
t = Thread(target=worker)
t.daemon = True
t.start()
# Tweepy sample based on Based on this:
# http://answers.oreilly.com/topic/2605-how-to-capture-tweets-in-real-time-with-twitters-streaming-api/
#
class StreamListener(tweepy.StreamListener):
    def on_status(self, status):
        print status.text
         # Adds the wave request to the queue.
       # Will throw an error if there is already a request
         # in the queue
        try:
              q.put_nowait('cat')
         except:
             print ''
def on_error(self, status_code):
        print >> sys.stderr, 'Encountered error with status code:', status_code
        return True # Don't kill the stream
def on_timeout(self):
        print >> sys.stderr, 'Timeout...'
        return True # Don't kill the stream
# Start the stream
sapi = tweepy.streaming.Stream(auth, StreamListener())
sapi.filter(track=[STREAM_FILTER] )

I ran into some threading issues combining the two APIs. By setting up a processing thread, and queue with a maximum depth of 1, I was able to ensure the Arduino was only handling one request at a time.

So here we go, the cat in action: There should have been a video here but your browser does not seem to support it.

The next step is to get myself an Ethernet shield for the Arduino, and maybe a web-cam.

Raspberry Pi Vs TDI

Tue, 13 Nov 2012 00:00:08 -1000

After waiting almost 14 weeks, my Raspberry Pi Model B finally arrived in the mail. I rushed down to the local Office Works, picked up a cheap SD card and a micro-USB charger, and then proceeded to download and install Raspbian; the Debian based distribution optimized for the Raspberry Pi. After some fiddling with the clear-case, and a lot of finger crossing, the machine booted up. So, now what?

When the Raspberry PI hit the shelves, one of things that I got really excited about was seeing it running WebSphere Application Server V8.5. Simone Maple put up a post showing a Raspberry PI running the new Liberty Profile. So it got me thinking; if it can run Websphere, can it run Tivoli Directory Integrator? TDI is a Java based integration product for quickly writing programs to move data from one point to another; think of it as a swiss-army-knife application.

The biggest hurdle is that the RaspberryPI runs an ARM chip, whereas the official TDI distribution is bundled with an Intel x86 compiled IBM JVM only. Unfortunately, IBM has not yet released an ARM compatible JVM, so I’d need to see if I could get the application running with an ARM compatible JVM. In this case, I selected OpenJDK, based on information in this post. Unfortunately, it doesn’t yet include JIT, so the performance will be less than ideal. The JVM can be installed using the standard debian package manager, using the command:

sudo apt-get install openjdk-6-jre libjna-java

So after installing Java, and checking that I could run it, I then attempted to install TDI, using the command line installation options:

sudo ./install_tdiv711_linux_x86.bin LAX_VM "/user/lib/jbm/java-1.6.0-openjdk-armhf/jre/bin/java"

I was hopeful that because they’d included an option to use a system JVM instead of the bundled JVM, that it might just work. One the installer roared to life, I selected only the TDI server component, and after waiting over an hour, the installation eventually completed.

So, would the server start? After fiddling with the JVM settings in the script file (tdiSetJavaHome.sh), I attempted to start the App Server using the following command:

./ibmdisrv -s ~/solutionDir -c ~/HelloWorld.xml -r Hello

After pondering the meaning of life, the application failed with a standard “Class Not Found” exception around the SSL handling, suggesting that the application was dependent on some JARs within the IBM JVM implementation. I fished the required JAR files out of a standard RedHat install of TDI (note: this is never recommended), and then cheekily copied them into the <INSERT_HERE_DIRECTORY>. I also went into the solution.properties file and disabled the SSL, system queue and system store settings to free up RAM. After starting it up again, and crossing all my fingers and toes, the server started.

My test application contained a basic TDI Assembly Line which consisted of HTTP Server component, and a small JavaScript component which calculates PI by iterating over a loop (based on this post). The script in my test example looped over 1 million times. I know its comparing apples and raspberry’s, but I wanted to see how the Raspberry PI compared to my standard work laptop (a Lenovo T410 with a i5 2.53GH CPU). My laptop could perform the loop in 242 milliseconds, whilst the Raspberry PI pumped out the same result in 51132 milliseconds. It doesn’t really prove a lot, and I’m assuming that the 512mb version would perform a little better. Once OpenJDK is optimised for the Raspberry PI, I’m sure the performance will improve.

Laptop Result

Raspberry PI Result

So, it’s not breaking any speed records, but it does prove the old Java adage: write once, run slowly everywhere.

Down the PhoneGap Rabbit Hole

Thu, 04 Oct 2012 00:00:08 -1000

I can still remember the day I picked up my first iPhone, a piano black 3G, the first iPhone they’d released in Australia. The little tike could do everything; surf the net, play 3D games, sync my contacts, make phone calls. I got all excited about writing my own apps for it, and then I just did nothing. The unwieldy Objective C syntax and the rather hectic day job whittled away all enthusiasm I had for building anything on the platform. Fast forward 4 years and the iOS development environment is a lot less prickly thanks to the introduction of development frameworks, and in particular, PhoneGap. PhoneGap allows you to write apps, for multiple mobile platforms, using good ol’fashioned HTML & Javascript instead of having to learn the underlying languages for each. Obviously there are tradeoffs, and they’re well documented (type “phonegap is ” into google), but the rapid development and multiplatform support should hopefully overcome any of the other foibles.

Apart from writing these tirades, I also co-run a bourgeoning Melbourne food blog. We’ve got a Wordpress site, with a custom theme I developed for the desktop, and a lighter one for mobile. It’s great, and I love running around taking photos of my food (much to the disgust of my dining companions) and I thought to myself that it would be great to expand in this area. I didn’t want a direct port of the website; that’s just useless; nobody uses those apps. So the goal is to make something people may actually use.

V0.1 — Proof of Concept

After spending a couple of nights sketching out the look and feel, I thought it would be time to see if I could translate it into HTML. Better to get a semi-functional proof of concept going than spend hours refining a design which ultimately doesn’t actually work on a phone. So using Eclipse as my editor, Bitnami MAPP as my app server stack, and the trusty Chrome Window Resizer, I translated my rather poor sketches into something that you could click through. Although both Chrome & Mobile Safari are Webkit at heart, there are some obvious differences that come into play, especially around speed. I’d made the innocent mistake of setting the initial-scale to 0.5 value, as a way of ensuring the page would look good on Retina devices. JQuery mobile didn’t like this, and so an animation that played perfectly well on my desktop, turned into a spluttering mess on the phone. Not really the kind of application experience I was looking for. So at the moment, I’m trying to iron out the JavaScript gremlins in the layout, using the Chrome extension Ripple. Ripple implements the PhoneGap APIs in Chrome, allowing easy debugging of web-apps written in PhoneGap (this remove the need to recompile the application after each change). Well, easier debugging, not easy debugging :)

Five Tivoli Identity Manager 5.1 Workflow Tips

Fri, 24 Aug 2012 00:00:08 -1000

The Tivoli Identity Manager 5.1 workflow engine can be punishing for beginners, so I’ve put together a list of handy hints to get you started. I spent hours banging my head into the desk working some of these out, but at least now you don’t have to!

1. External Script Files for Development

Based on the glorious suggestion from this Stephen Swann’s blog, it’s so much easier to externalise your Javascript nodes during development. It means you don’t need to interact with the workflow editor as much, and you can quickly test your changes. I’ve attached below a sample Javascript function which reads in an external Javascript file from the filesystem, and then executes it.

function externalWorkflow(fileName)
{
   // Systems new line component
   var newLineChar = java.lang.System.getProperty("line.separator");
 // Open the external JavaScript file
    var scriptFile = new java.io.BufferedReader(new java.io.FileReader(fileName));
  var lineOfCode = null; var code = "";
  // Read the file and add each line of code we find to a variable.
  // Needs to append the new line character which the readLine function is removing.
  while ((lineOfCode = scriptFile.readLine()) != null)
    { code += lineOfCode + newLineChar; }
   // Close the file scriptFile.close();
   // Execute the code that we have gathered together
  eval(code);
}
externalWorkflow("D:/Workflows_KE/A20010_parseRequest.js");

2. POJO is the new Black

The IBM JSEngine allows you to access plain old Java objects (POJOs) from within your workflow. This allows you to reuse your existing Java classes with your TIM workflows, and it allows you to access some of the more advanced Java features, such Java XML bindings. It’s important, however, not to use Java as a crutch, particularly if you come from a Java background. It’s sometimes very tempting to jump into banging out Java classes, when the same outcome can be achieved a little differently with a lot less Javascript. With great power comes great responsibility.

3. JS Objects != Java Objects

I learnt this lesson the hard way. Although TIM 5.1 has the ability to interact with Java objects (by referencing them in the scriptframework.properties file), the implementation of the IBM JSEngine appears to have its own “String” object. The following script example, throws a “Script interpreter error” because the IBM JSEngine doesn’t implement the getClass() method:

var example1 = new java.lang.Integer(0);
var example2 = new java.lang.String("Hello");
activity.auditEvent("Integer: " + example1.getClass());

activity.auditEvent("String: " + example2.getClass());

So be careful with your String manipulation, you never know what’s missing.

4. Consistent Logging

One thing that really bugs me (pun intended) when reviewing workflow code, is the inconsistent use of TIMs logging functions. There are two approaches to logging, and both have their place.

Enrole.log() — An error message logged to the msg.log file. Great for detailed technical errors.
Activity.auditEvent() — As the name suggests, a great way of logging audit events which are visible in the GUI. Log events like state changes, and email destinations, not stack traces.

5. Automated Unit Testing

As I mentioned in my previous post, it always pays to automate your workflow unit testing as much as possible. It can be very tedious manually entering a lot of data into TIM, so if you can automate it then you’re a step closer to making your life a lot easier. A lot of TIM developers gloss over this, and it shows when it comes to QA

Unit Testing Tivoli Identity Manager Workflows

Tue, 24 Jul 2012 00:00:08 -1000

At work, I’ve recently dabbled in the mysterious art of Tivoli Identity Manager (TIM) workflow development. TIM is an extremely flexible identity lifecycle management product, and part of this flexibility is due to the ability to write custom workflows with Javascript. However, the ability to write Javascript workflows is a double-edged sword; the product makes it very difficult to automate any unit testing of the workflows. Luckily, automated unit testing can be easily accomplished by leveraging a couple of different software packages.

We’re going to use a combination of the IBM Tivoli Identity Manager API Web Services Wrappers, an unsupported web-services interface which exposes the TIM Java API via SOAP, and we’re going to invoke it using SOAP-UI, an open-source testing tool which can invoke web-services. A match made in heaven.

The download and installation of the TIM Wrapper is straightforward, and it just sits within the same application server instance as TIM. Once you’ve installed the application, you can test that the wrapper is available by retrieving its WSDL through a web-browser http://<host name>:9080/ITIMWebServices/services/WSItimService?WSDL

Once you’ve confirmed that the WSDL is accessible, it’s time to fire up SOAP-UI. Right click on “Projects”, and then select “New soapUI Project”. Add a new project name in, and enter the WSDL tested above. A new project should be created with all the operations. The WSItimService WSDL contains most of the TIM API, but not all of it; you might need to use one of the other services offered by the wrapper to access all of the API operations. Now we’re ready to setup our unit test extravaganza.

Right click on the project and click “New Test Suite”, and then enter a suitable name. The test suite can contain one or more test cases. Your new test suite will appear, and then right click within the white space of the dialogue and click “New TestCase”, and then enter “TestCase1” as the name. The following test case will appear:

The test case dialogue allows you to specify a set of steps to validate your test case. For our sample test case, we’re going to create a new person object in TIM, and then validate that it’s been successfully created.

The first step is to login to TIM. Almost all requests into TIM require a valid session, which can be established using the “WSItimServiceSoapBinding -> login” request. Click the “SOAP” labeled button within the testcase, and enter “Establish Session” in the name. A list of valid requests will then appear; scroll down and select the “WSItimServiceSoapBinding -> login” request. Click “Ok”. An “Establish Session” dialogue will now appear, containing a template SOAP message. To enable login, you’ll need to enter in a valid TIM user ID in the “principal” element, and the user’s password in the “credential” element. For example:

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ser="http://services.ws.itim.ibm.com">
   <soapenv:Header/>
   <soapenv:Body>
      <ser:login>
         <ser:principal>itim manager</ser:principal>
         <ser:credential>superpassword</ser:credential>
      </ser:login>
   </soapenv:Body>
</soapenv:Envelope>

To test that you’ve done it correctly, click the green submit request button. If all has gone to plan, you should see a rather lengthy response, containing a “clientSession” and a “sessionID”. These two values are used in subsequent requests to maintain the session.

The next step is to create a person in TIM. Click the “SOAP” button again, but this time select the “WSItimServiceSoapBinding -> createPerson” request. The XML for this request is a bit more complicated, so I’ve mocked up a sample below. Copy the following piece of XML into the request field:

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
   <soapenv:Body>
     <!-- This is a createPerson request -->
      <createPerson xmlns="http://services.ws.itim.ibm.com">
           <!-- The information within this session element is retrieved from the login response. We'll talk about setting these values later on-->
         <session>
            <ns1:clientSession xmlns:ns1="http://model.ws.itim.ibm.com">?????</ns1:clientSession>
            <content:enforceChallengeResponse xmlns:content="http://model.ws.itim.ibm.com">false</content:enforceChallengeResponse>
            <excerpt:locale xmlns:excerpt="http://model.ws.itim.ibm.com">
               <excerpt:country>AU</excerpt:country>
               <excerpt:language>en</excerpt:language>
               <excerpt:variant/>
            </excerpt:locale>
            <ns4:sessionID xmlns:ns4="http://model.ws.itim.ibm.com">?????</ns4:sessionID>
         </session>
      <!-- This is the container where the person will be created. This is a very simple TIM install, so we're just creating the person at the base ou. -->
         <wsContainer>
            <ns6:itimDN xmlns:ns6="http://model.ws.itim.ibm.com">erglobalid=00000000000000000000,ou=dev, DC=COM</ns6:itimDN>
            <ns8:select xmlns:ns8="http://model.ws.itim.ibm.com">false</ns8:select>
            <ns11:profileName xmlns:ns11="http://model.ws.itim.ibm.com">OrganizationalUnit</ns11:profileName>
         </wsContainer>
      <!-- This secion sets the attributes for the person being created. Again, this is a very simple person example. If you're creating a more complex example,
       it's usually easier to create a sample user through the GUI, and then lookup that person using an LDAP browser to determine the various attribute value. -->
         <wsPerson>
            <ns13:attributes xmlns:ns13="http://model.ws.itim.ibm.com">
               <item>
                 <!-- This sets the preferred UID for the user. The ${ } brackets contain a piece of Groovy code which will be executed before the request is sent to the server.
                     This piece of code appends a randomyl generated number to the UID.-->
                  <ns13:name>uid</ns13:name>
                  <ns13:values>
                     <item>UIDOne${=(int)(Math.random()*9999999)}</item>
                  </ns13:values>
               </item>
               <item>
                  <ns13:name>sn</ns13:name>
                  <ns13:values>
                     <item>Lee${=(int)(Math.random()*9999999)}</item>
                  </ns13:values>
               </item>
               <item>
                  <ns13:name>givenName</ns13:name>
                  <ns13:values>
                     <item>Peggy</item>
                  </ns13:values>
               </item>
<item>
                  <ns13:name>cn</ns13:name>
                  <ns13:values>
                     <item>Test CN</item>
                  </ns13:values>
               </item>
            </ns13:attributes>
            <ns15:name xmlns:ns15="http://model.ws.itim.ibm.com">UIDOne${=(int)(Math.random()*9999999)}</ns15:name>
            <ns16:select xmlns:ns16="http://model.ws.itim.ibm.com">false</ns16:select>
            <!-- The type of object being created, in this case, a person. -->
            <ns17:profileName xmlns:ns17="http://model.ws.itim.ibm.com">Person</ns17:profileName>
         </wsPerson>
         <date xsi:type="xsd:dateTime">2012-02-29T00:38:35.111Z</date>
      </createPerson>
   </soapenv:Body>
</soapenv:Envelope>

This piece of XML code however, is incomplete. We need to copy the session values from the “WSItimServiceSoapBinding -> login” response, into this new request. Luckily, SOAPUI has a “Property Transfer” function to copy the value into the second request. Go back to the Test Case view, and click the “Property Transfer” button and then specify a name (ie Session Transfer).

A session transfer dialogue will appear. Click the “Plus” button in the corner, and enter the name “ClientSession”. A dialogue will appear with “source” and “target” panels. In the “Source” panel, ensure “Establish Session” is selected, and the property is set to “Response”. Then set the following XPATH within the textbox:

declare namespace  soapenv="http://schemas.xmlsoap.org/soap/envelope/";
declare namespace ser="http://services.ws.itim.ibm.com";
//soapenv:Envelope/soapenv:Body/ser:loginResponse/ser:loginReturn/ser:clientSession

In the “Target” panel, ensure “Create Test Person” is selected, and the property is set to “Request”. Then set the following XPATH within the textbox:

declare namespace soapenv=“http://schemas.xmlsoap.org/soap/envelope/”; declare namespace ser=“http://services.ws.itim.ibm.com”;

declare namespace mod=“http://model.ws.itim.ibm.com”; //soapenv:Envelope/soapenv:Body/ser:createPerson/ser:session/mod:clientSession

Click “Run” to execute the XPATH. The client session value will now be copied from the “Login” response to the “createPerson” request. This also needs to be performed for the Session ID. Use the following XPATHS.

Source

declare namespace soapenv="http://schemas.xmlsoap.org/soap/envelope/"; declare namespace ser="http://services.ws.itim.ibm.com"; //soapenv:Envelope/soapenv:Body/ser:loginResponse/ser:loginReturn/ser:sessionID

Target

declare namespace soapenv="http://schemas.xmlsoap.org/soap/envelope/"; declare namespace ser="http://services.ws.itim.ibm.com"; declare namespace mod="http://model.ws.itim.ibm.com"; //soapenv:Envelope/soapenv:Body/ser:createPerson/ser:session/mod:sessionID

Now that we have the session information, we should be able to execute our first run of the test case. Go back to the test case dialogue, and click “Run”. If all goes well, you should see a green FINISHED graphic. You should be able to see the Person in the TIM Pending requests view.

You may have noticed that although we can now successfully create a person in TIM, we’re not actually verifying that the person is being successfully created. TIM is asynchronous, so the response to the “createPerson” request is a “requestID” and a status that states that the request is in a “not started” state.

What we really want to know is if the person has been successfully created without checking through the GUI. To do this, we’re going to use a Loop within the TestCase to periodically poll TIM to check the response status. Once the request has been processed, we’ll then programmatically evaluate whether it was successful. Manual verification be gone! Create a new SOAP message called “Get Request Wait” for the ““WSItimServiceSoapBinding -> getRequest” request. Set the body as follows:

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ser="http://services.ws.itim.ibm.com" xmlns:mod="http://model.ws.itim.ibm.com">
   <soapenv:Header/>
   <soapenv:Body>
      <ser:getRequest>
         <ser:session>
            <mod:clientSession>?</mod:clientSession>
            <mod:enforceChallengeResponse>false</mod:enforceChallengeResponse>
            <mod:locale>
               <mod:country>AU</mod:country>
               <mod:language>en</mod:language>
            </mod:locale>
            <mod:sessionID>?</mod:sessionID>
         </ser:session>
     <!-- This is where the request ID will be set by the property transfer -->
         <ser:requestId>?</ser:requestId>
      </ser:getRequest>
   </soapenv:Body>
</soapenv:Envelope>

We need to create an additional SOAP message, using the same syntax as above, but it needs to be called “Get Request Final”. This is the message we’ll eventually use to verify the completion status. Before we do that, create a new “Property transfer” called “Get Request Transfer” above the first “Get Request” message. Create the following transfers within this:

Transfer: ClientSession
Source: Establish Session
Source Property: Response
Source XPATH:
declare namespace soapenv="http://schemas.xmlsoap.org/soap/envelope/";
declare namespace ser="http://services.ws.itim.ibm.com";
//soapenv:Envelope/soapenv:Body/ser:loginResponse/ser:loginReturn/ser:clientSession
Target: Get Request Wait
Target Property: Request
Target XPATH:
declare namespace soapenv="http://schemas.xmlsoap.org/soap/envelope/";
declare namespace ser="http://services.ws.itim.ibm.com";
declare namespace mod="http://model.ws.itim.ibm.com";
//soapenv:Envelope/soapenv:Body/ser:getRequest/ser:session/mod:clientSession
Transfer: SessionID
Source: Establish Session
Source Property: Response
Source XPATH:
declare namespace soapenv="http://schemas.xmlsoap.org/soap/envelope/";
declare namespace ser="http://services.ws.itim.ibm.com";
//soapenv:Envelope/soapenv:Body/ser:loginResponse/ser:loginReturn/ser:sessionID
Target: Get Request Wait
Target Property: Request
Target XPATH:
declare namespace soapenv="http://schemas.xmlsoap.org/soap/envelope/";
declare namespace ser="http://services.ws.itim.ibm.com";
declare namespace mod="http://model.ws.itim.ibm.com";
//soapenv:Envelope/soapenv:Body/ser:getRequest/ser:session/mod:clientSession
Transfer: RequestID
Source: Create Test Person
Source Property: Response
Source XPATH:
declare namespace soapenv="http://schemas.xmlsoap.org/soap/envelope/";
declare namespace ser="http://services.ws.itim.ibm.com";
declare namespace mod="http://model.ws.itim.ibm.com";
//soapenv:Envelope/soapenv:Body/ser:createPersonResponse/ser:createPersonReturn/ser:requestId
Target: Get Request Wait
Target Property: Request
Target XPATH:
declare namespace soapenv="http://schemas.xmlsoap.org/soap/envelope/";
declare namespace ser="http://services.ws.itim.ibm.com";
declare namespace mod="http://model.ws.itim.ibm.com";
//soapenv:Envelope/soapenv:Body/ser:getRequest/ser:requestId
Transfer: ClientSession_final
Source: Establish Session
Source Property: Response
Source XPATH:
declare namespace soapenv="http://schemas.xmlsoap.org/soap/envelope/";
declare namespace ser="http://services.ws.itim.ibm.com";
//soapenv:Envelope/soapenv:Body/ser:loginResponse/ser:loginReturn/ser:clientSession
Target: Get Request Final
Target Property: Request
Target XPATH:
declare namespace soapenv="http://schemas.xmlsoap.org/soap/envelope/";
declare namespace ser="http://services.ws.itim.ibm.com";
declare namespace mod="http://model.ws.itim.ibm.com";
//soapenv:Envelope/soapenv:Body/ser:getRequest/ser:session/mod:clientSession
Transfer: SessionID_final
Source: Establish Session
Source Property: Response
Source XPATH:
declare namespace soapenv="http://schemas.xmlsoap.org/soap/envelope/";
declare namespace ser="http://services.ws.itim.ibm.com";
//soapenv:Envelope/soapenv:Body/ser:loginResponse/ser:loginReturn/ser:sessionID
Target: Get Request Final
Target Property: Request
Target XPATH:
declare namespace soapenv="http://schemas.xmlsoap.org/soap/envelope/";
declare namespace ser="http://services.ws.itim.ibm.com";
declare namespace mod="http://model.ws.itim.ibm.com";
//soapenv:Envelope/soapenv:Body/ser:getRequest/ser:session/mod:clientSession
Transfer: RequestID_final
Source: Create Test Person
Source Property: Response
Source XPATH:
declare namespace soapenv="http://schemas.xmlsoap.org/soap/envelope/";
declare namespace ser="http://services.ws.itim.ibm.com";
declare namespace mod="http://model.ws.itim.ibm.com";
//soapenv:Envelope/soapenv:Body/ser:createPersonResponse/ser:createPersonReturn/ser:requestId
Target: Get Request Final
Target Property: Request
Target XPATH:
declare namespace soapenv="http://schemas.xmlsoap.org/soap/envelope/";
declare namespace ser="http://services.ws.itim.ibm.com";
declare namespace mod="http://model.ws.itim.ibm.com";
//soapenv:Envelope/soapenv:Body/ser:getRequest/ser:requestId

Now we need to establish some loop logic. Create a new Property Transfer called “Loop Status” between “Get Request Wait” and “Get Request Final”. Now, click on the test case name, “TestCase1” in my case, and then click the “Test Properties” tab in the lower left hand corner. Add a new property called “getRequestStatus”. Back in the “Loop Status” property transfer, create the following transfer:

Transfer: Job Status
Source: Get Request Wait
Source Property: Response
Source XPATH:
declare namespace soapenv="http://schemas.xmlsoap.org/soap/envelope/";
declare namespace ser="http://services.ws.itim.ibm.com";
declare namespace mod="http://model.ws.itim.ibm.com";
//soapenv:Envelope/soapenv:Body/ser:getRequestResponse/ser:getRequestReturn/ser:processStateString
Target: TestCase1
Target Property: getRequestStatus

This transfer uses XPATH to copy the request status into the “getRequestStatus” property/variable. We now need a piece of Groovy code to evaluate the status. Within the TestCase, add a new “Groovy Script” below “Loop Status”, called “While Loop”. Within the script, copy the following Groovy script, which will act as our loop logic.

//This retrieves the status from the testCase property
status = testRunner.testCase.getPropertyValue("getRequestStatus")

//Prints out the job status
log.info "Job Status: " + status;
/*If the record hasn't processed yet, loop back around.
These are the various TIM5.1 processing states. If it is one of
these states, go back up to the Get Request Wait Step. Repeat the loop
until it has finished processing.
*/
if(status == "In Process" || status == "Running" || status == "Not Started")
{
 //Sleep for 10 seconds.
 Thread.sleep(10000);
    //Loop back to the request
  testRunner.gotoStepByName("Get Request Wait");
}

The above loop logic will exit out when the request state, retrieved by the “Get Request Wait” request, isn’t in the state “In Process”, “Running” or “Not Started”. The key now is to verify whether the request state, once it has finished processing, is successful. We can do this using an assertion on the “Get Request Final” request. Assertions in SOAPUI allow you to apply rules against responses to validate that the response received is correct. Open the “Get Request Final” request, and then click the “Assertions” button in the bottom left corner of the dialogue.

Add a new insertion, called “XPATH Match”, and enter the following values into the dialogue:

declare namespace soapenv="http://schemas.xmlsoap.org/soap/envelope/";
declare namespace ser="http://services.ws.itim.ibm.com";
declare namespace mod="http://model.ws.itim.ibm.com";
//soapenv:Envelope/soapenv:Body/ser:getRequestResponse/ser:getRequestReturn/ser:statusString

Expected Result:
Succeeded

This assertion will check that the response received from TIM is successful. Now that the assertion is in place, we can go back to the test case and run it. This code will now establish a session with TIM, create a person, and then verify that the person has been successfully created. Although this example is fairly basic, this kind of automated unit testing can significantly decrease the amount of time spent developing and verifying TIM Workflow code, as we can create new objects in a matter of seconds, instead of having to tediously do it through the web-interface. This same logic can be applied to not just “Person” objects, but to anything that can be created through the API. The workflow sky is really the limit..

To get you started, a standalone SOAP-UI project containing this example can be downloaded from Github.

Hadoop & HBase — Hello World

Sun, 08 Jul 2012 00:00:08 -1000

Hadoop & HBase — Hello World

Ever since seeing the picture of the cute little stuffed elephant on the page, I was excited about Hadoop. I was also excited about the potential to quickly build large-scale, distributed applications, but I was mainly excited about the elephant.

Harnessing that enthusiasm, this post will document how to build a venerable “hello world” example using Hadoop & Hbase. Hbaseis a non-relational, distributed database modeled after Google’s BigTable. Hbase runs on top of Hadoop, which, in this example provides a distributed file system where HBase’s data sits.

Step 1: Install the Operating System in a VM

First off, I’ve downloaded Ubuntu 12.04 Server, 64 Bit edition, and installed it in a Virtual Machine. I’ve used VMware Workstation 8.0.3, but VirtualBox will work just as well. You’ll also want to install a text editor, such as VIM (here’s a tutorial for those who haven’t used it before):

sudo apt-get install vim

Step 2: Modify Hosts File

I ran into a number of unusual issues initially when trying to get HBase to start, and they were caused by an incorrectly configured /etc/hosts file. I would have saved myself hours if I’d actually read the Hbase documentation. Open the /etc/hosts file using:

sudo vim /etc/hosts

Change the following host mapping, from:

127.0.0.1 localhost

127.0.1.1 ubuntu

to this:

127.0.0.1 localhost

<machineIPAddress> Ubuntu

where <machineIPAddress> is the IP of the Virtual Machine. The Virtual Machine in this case is called “Ubuntu”. You can use iconfig on the VM to get your VMs IP Address.

In addition, you’ll need to modify the host file entry on your local machine so it can resolve the hostname of the Virtual Machine.

Step 3: Install Oracle Java Implementation

Hadoop is built and tested with the Oracle JVM, which is surprisingly a little difficult to install on Ubuntu. The easiest option is to use a prepackaged version; I ended up using this Duinsoft script to install it. Open the file /etc/apt/sources.list using:

sudo vim /etc/apt/sources.list

And then add the following item to the end of the file:

deb http://www.duinsoft.nl/pkg debs all

The next step is to import the key for the repository:

sudo apt-key adv — keyserver keys.gnupg.net — recv-keys 5CB26B26

Finally, install Java using the following two commands:

sudo apt-get update

sudo apt-get install update-sun-jre

Step 4: Install SSH

The next step is to install an SSH server; this is required to run Hadoop. I also find it a little easier to interact with the Virtual Machine through an SSH session, rather than through the VM interface. To do this, run:

sudo apt-get install openssh-server

Step 5: Hadoop User

Once the SSH server has been installed, a new user needs to be created to run Hadoop. To do this, run:

sudo addgroup hadoop

sudo adduser — ingroup hadoop huser

The huser needs to have a public key generated to allow it to connect to the server without a password. The first step is to login as the user:

sudo -i -u huser

Then generate the key, and add it to the allowed list of keys which can be used to connect:

ssh-keygen -t dsa -P ‘’ -f ~/.ssh/id_dsa

cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

Step 6: Download Hadoop

It’s now time to download the Hadoop Common package. Enter the following to download the 1.0.3 release to the husers home directory

cd ~

wget http://mirror.overthewire.com.au/pub/apache/hadoop/common/hadoop-1.0.3/hadoop-1.0.3.tar.gz

Once the file has downloaded, untar it with the following command:

tar -zxvf hadoop-1.0.3.tar.gz

A folder called hadoop-1.0.3 will now be in the home directory. Unfortunately the intended destination is protected, so you’ll need to use a user with sudo access to move the download. To do this, enter the following:

exit

sudo mv /home/huser/hadoop-1.0.3 /usr/local/hadoop

sudo -i -u huser

It’s now time to configure!

Step 7: Configure Hadoop

The newly copied Hadoop installation needs to have its JAVA_HOME variables set. To do this, navigate to the directory:

cd /usr/local/hadoop/conf

And then open the file hadoop-env.sh using the following command. This file contains some environment variable settings used by Hadoop:

vim hadoop-env.sh

Once the file is open, append the following line to it (you will need to update this depending on the java version installed):

export JAVA_HOME=/opt/java/32/jre1.6.0_31

Save the file.

The next important configuration step is to setup Hadoop. We’re going to be running Hadoop in a single-node setup. This is a very basic configuration, but it’s good for development. The first step is to create a folder on the filesystem for Hadoop to store its files:

mkdir /home/huser/hadoop_tmp

Once done, navigate to /usr/local/hbase/conf, and open up the file core-site.xml, and ensure the contents of it look like this:

<?xml version=”1.0”?>
 <?xml-stylesheet type=”text/xsl” href=”configuration.xsl”?>
 <configuration>
 <property>
 <name>hadoop.tmp.dir</name>
 <value>/home/huser/hadoop_tmp</value>
 </property>
 <property>
 <name>fs.default.name</name>
 <value>hdfs://ubuntu:8020</value>
 </property>
 </configuration>

The value for “fs.default.name” is the name of the default file system, and the value for “hadoop.tmp.dir” is the base for temporary files on the local file system. Once you’re done, save the file.

The Hadoop filesystem needs to be formatted with the new settings. To do this, execute:

/usr/local/hadoop/bin/hadoop namenode –format

Once done, the instance can be started with the following command:

/usr/local/hadoop/bin/start-all.sh

The Hadoop instance should now be running. To confirm this, go back to your host machine, and navigate to the following page in a web-browser

http://ubuntu:50070/dfshealth.jsp

Step 8: Adding a Hbase Directory in Hadoop

A directory needs to be made within the Hadoop filesystem to store the Hbase data. To do this, execute the following:

/usr/local/hadoop/bin/hadoop fs -mkdir hbase_dir

A directory now exists on the Hadoop file system, called /user/huser/hbase_dir, you should be able to see it through the web interface.

Step 9: Download HBase

The next step is to download HBase. Enter the following to download the 1.0.3 release to the husers home directory

cd ~

wget http://mirror.mel.bkb.net.au/pub/apache/hbase/hbase-0.94.0/hbase-0.94.0.tar.gz

Once the file has downloaded, untar it with the following command:

tar -zxvf hbase-0.94.0.tar.gz

A folder called hbase-0.94.0 will now be in the home directory. Again the intended destination is protected, so you’ll need to use a user with sudo access to move the download. To do this, enter the following:

exit

sudo mv /home/huser/hbase-0.94.0 /usr/local/hbase

sudo -i -u huser

Step 10: Configure Hbase

The newly copied Hbase installation needs to have its JAVA_HOME variables set. To do this, navigate to the directory:

cd /usr/local/hbase/conf

And then open the file hbase-env.sh using the following command. This file contains some environment variable settings used by Hadoop:

vim hadoop-env.sh

One the file is open, append the following line to it (you will need to update this depending on the java version)

export JAVA_HOME=/opt/java/32/jre1.6.0_31

Save the file.

The next step is to define the hbase setup. To do this, open the file using:

vim hbase-site.xml

And then ensure the contents of the file appear as follows:

<configuration> 
 <property> 
 <name>hbase.rootdir</name> 
 <value>hdfs://ubuntu:8020/user/huser/hbase_dir</value> 
 <description> 
 </description> 
 </property> 
 <property> 
 <name>hbase.master</name> 
 <value>ubuntu:60000</value> 
 <description> 
 </description> 
 </property> 
 </configuration>

Now it’s all configured, it’s time to start Hbase up. To do this, execute:

/usr/local/hbase/bin/start-hbase.sh

The application should now be running. To confirm this, go back to your host machine, and navigate to the following page in a web-browser:

http://ubuntu:60010/master-status

Step 11: Creating Your First Table

HBase contains its own shell, where you can interact with it. To start the shell, execute:

/usr/local/hbase/bin/hbase shell

Within the shell, enter the following command to create a new table called “words”, with a column family called “definitions”:

create ‘words’,’definitions’

The next step will be to load up some data. Obviously, you’d generally do this with your application, but for this tutorial we’ll used fixed data. Execute the following, one line at a time.

put ‘words’,’hello’,’definitions:1’,’A greeting (salutation) said when meeting someone or acknowledging someone’s arrival or presence.’

put ‘words’,’hello’,’definitions:2’,’A greeting used when answering the telephone.’

put ‘words’,’world’,’definitions:1’,’to consider or cause to be considered from a global perspective’

These commands insert two rows into the database, two for “hello”, and one for “world”. The “hello” row has two different columns within the “definitions” column family.

Step 12: Finally, Program Time

So all the prep work, is done, it’s time to run a sample program. I’ve attached below a simple Java class which queries the rows within this table and prints them out to the screen.

package au.com.gigliotti.sandpit.hbase;

import java.util.NavigableMap;
import java.util.NavigableSet;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.client.HBaseAdmin;
import org.apache.hadoop.hbase.client.HTableFactory;
import org.apache.hadoop.hbase.client.HTableInterface;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.client.ResultScanner;
import org.apache.hadoop.hbase.client.Scan;
import org.apache.hadoop.hbase.util.Bytes;

/**
 * Sample HBase application which retrieves all rows from a table and then
 * prints them to the console. The table definition is as follows:
 *
 * table: words column family: definitions
 *
 * @author Gerard Gigliotti (gerard.gigliotti.com.au)
 *
 *
 */
public class HBaseSample {

// The name of the table being queried
 public static final byte[] WORDS_TABLE = Bytes.toBytes("words");
  // The column family
    public static final byte[] DEFINITIONS_FAMILY = Bytes
           .toBytes("definitions");

/**
    * Construct creates a new instance of HBaseSample, and invokes the start
    * method.
   */
 public static void main(String[] args) throws Exception {
       (new HBaseSample()).start();
    }

/**
     * Method prints out all rows within the words table.
    * */
   private void start() throws Exception {

// Load's the hbase-site.xml config
      Configuration config = HBaseConfiguration.create();
     HTableFactory factory = new HTableFactory();
        HBaseAdmin.checkHBaseAvailable(config);

// Link to table
     HTableInterface table = factory.createHTableInterface(config,
               WORDS_TABLE);

// Used to retrieve rows from the table
        Scan scan = new Scan();
// Scan through each row in the table
        ResultScanner rs = table.getScanner(scan);
      try {
           // Loop through each retrieved row
          for (Result r = rs.next(); r != null; r = rs.next()) {
              System.out.println("Key: " + new String(r.getRow()));
             /*
               * Cycles through each qualifier within the "definitions" family.
              * The column family, with the qualifier, make up the column
                 * name; it is usually represented in the syntax
                 * family:qualifier. In our example, each qualifier is a number.
                 */
             NavigableMap familyMap = r
                      .getFamilyMap(DEFINITIONS_FAMILY);
              // This is a list of the qualifier keys
             NavigableSet keySet = familyMap.navigableKeySet();
// Print out each value within each qualifier
             for (byte[] key : keySet) {
                 System.out.println("t Definition: " + (new String(key))
                           + ", Value:"
                          + new String(r.getValue(DEFINITIONS_FAMILY, key)));
             }
           }
       } catch (Exception e) {
         throw e;
        } finally {
         rs.close();
     }

As you’ll have noticed, the connection settings are not defined within the class. A separate file, called hbase-site.xml, which needs to be added to your classpath, contains the settings for connecting to Hbase.

The code can be downloaded from GitHub here.

That’s it; you’ll now have your very own singing and dancing Hadoop & Hbase instance.

References

The HBase documentation is very useful: http://hbase.apache.org/book.html
A great post about establishing a Hadoop instance: http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/
Another great post, but focusing more on Hbase: http://ria101.wordpress.com/2010/01/28/setup-hbase-in-pseudo-distributed-mode-and-connect-java-client/

CSC Ghana #8 Final Post

Mon, 28 May 2012 00:00:08 -1000

I thought a good way of finishing up my Corporate Service Corps (CSC) Ghana #8 assignment would be to answer the following set of interview questions; I was asked these during our final interview.

What did CSC participation mean to you?

To me it was an honour to be selected to represent IBM on such a critical assignment for the people of Ghana. It was a privilege to work with such an exemplary group of IBMers from around the world; we were able to leverage each other’s unique abilities to deliver a quality set of deliverables in a very short time.

Tell us a little about your assignment.

On our assignment we focused on developing a supply chain IT roadmap for the Ministry of Health (MoH). We analysed the existing supply chains system, and then made a number of recommendations for improving their approach by leveraging new IT systems.

What surprised you about your experiences in Ghana?

The thing that surprised me the most was how willing everyone was to help. People are sometimes weary of consultants, but there was a genuine desire within the MoH to fix their supply chain issues and deliver better health outcomes for the people of Ghana.

What do you think you learned/gained from the experience?

I really gained a greater appreciation for the challenges that you can face when working within a multicultural team. My assignment team had people from the US, China, India and Australia, and it was interesting to see the differences in how each team member approached their work. Even something as simple as spelling can be an issue. For example, a number of words in Australian English are spelt with an “s” (ie, organise), but in US English they are spelt with a “z” (organize).

What will you take back to your day job?

I’ll bring back to my day job a global network of IBM contacts which I’ll be able to leverage to help deliver better outcomes for our clients.

Another team from IBM will be coming in the fall. Any words of advice?

Just a little list of things:

Make sure you bring over a Visa card for ATM withdrawals. MasterCard isn’t widely accepted.
Try Grasscutter, a local delicacy. Country Kitchen, in Accra, occasionally has it as a special. I suggest not buying it from the roadside.
If you can only do one weekend activity, make sure it’s the Dodi Princess Cruise.
Bring some food from your country of origin to share with the team. Some of the best moments on the trip centred on these exchanges.

That’s all for Ghana #8. It’s been emotional.

Press Release

Thu, 24 May 2012 00:00:08 -1000

As the project finished up, a press release was issued documenting our work. A couple of different news agencies chose to cover the story:

Dancing troupe at the Mountain Gorillas View Hotel

Tue, 22 May 2012 00:00:08 -1000

There should have been a video here but your browser does not seem to support it.

Dancing troupe at the Mountain Gorillas View Hotel. I was the only audience member for 25 minutes.

Barrel of Monkeys

Mon, 21 May 2012 00:00:08 -1000

I was too smug about the trekking yesterday, and I was made to pay for it today. Due to the expense of the gorilla permits ($500 a day, rising to $750 in July), I decided to visit the Golden Monkeys on my second day; another rare attraction limited to the Parc National des Volcans and its neighbouring parks. This trek was supposed to be easier than the gorillas, and I was looking forward to a leisurely walk around the peripheries of the park. Oh I was so wrong.

As we headed out from our drop off point, it started to rain. Not that slow-motion-romantic-rain which occurs when you’re rushing towards a loved one, but the now-you’re-in-the-tropics-boy kind of rain. It bucketed down in a manner similar to the end sequence of the movie 2012. Stupidly, I had left my raincoat in Australia but I had luckily picked one up at Johannesburg airport. For future reference, yes, the $40 pocket raincoat is not as effective as a $400 gor-tex one. Surprisingly there is a reason that outdoor adventure gear costs so much. Well, maybe not that much. My camera bag, even under my top, was soaked, but the camera was only a little damp and still functioning. The rain turned all the ground to mud, making dodging thistles, vines and ankle twisting rocks that much harder. We eventually located the tracker group, and then finally a group of Golden Monkeys. Unlike the Gorillas, which happily sit there and let you photograph them, the monkeys crazily jump all over the place. Amazingly, the rain stopped for a good half an hour, allowing me to take some truly appalling photos. I’m blaming the monkeys.

After 45 minutes, the heavens opened up again, and we decided to head back. The forest by now resembled a muddy river, and it was so deep in parts that we had to wade through it. Nobody was safe from the downpour, and I was so soaked that water was pouring out of my sleeves like a spout. Bear Grylls eat your heart out.

Gorillas in the Mist

Sun, 20 May 2012 00:00:08 -1000

The journey to Rwanda was excruciatingly tiring. Starting in Kumasi, we had to drive 6 hours back to Accra, where I had to catch a 5 hour flight to Johannesburg, which left at 11pm and arrived at 6am (it’s in a different time zone). I then had a 5 hour stopover at the airport, and then another 4 hour flight to Kigali, via Bujumbura. The entire experience was surreal; the second flight wasn’t announced at the airport, wasn’t on the departure board, and there were only 20 people on a jet with seating for 150.

On arrival, immigration lovingly gave me a 50% cheaper visa than I was expecting, and my bag miraculously appeared on the carousel. Outside, my guide John was waiting for me with a little sign with my name on it. Tick that off my list of things to do. John then told me that I was the only person on the tour; 4 days of John and me.

First stop was a rather deserved rest at the Lemigo Hotel in Kigali. I ended up getting a club sandwich from room service, and it only cost me 5000 Rwandan Francs ($8 AUD). The next day, John was scheduled to pick me up at 2:00pm, so I decided to walk around the neighbourhood near the hotel. I ended up at Great Wall Chinese Restaurant, which served up this rather interesting interpretation of sweat and sour pork. Avert your eyes if you’re easily offended.

At 2:00pm, we made our way to the Mountain Gorilla View Lodge, though the magnificent Rwandan countryside. I can’t explain how pretty Rwanda is as a country; everywhere you look there is a view more beautiful than the last. The lodge itself was fantastic, with very attentive staff. The fire place in my room was kept burning all afternoon and well into the evening, and someone else delivered a hot water bottle before I went to sleep.

I did not enjoy the 5:30am wake up call for the Gorilla trekking, but the excitement of getting to see a gorilla in the wild soon overtook the sleepiness. John and I had a quick trip to the Parc National des Volcans headquarters, where I was assigned to see the Agashya Group, which is the second largest group, and one of the easier to locate. We drove out to the starting point, and then after some 30 minutes of trekking through some rather muddy forest, we located the group. They were almost all sitting down eating.

After about 20 minutes, the giant silverback decided he wanted something else to eat, so he headed off. This is where things got interesting, as we followed the group through the forest with the guides cutting a path through the thick vegetation.

All too soon the magical hour was over. It was a privilege being able to see such magnificent creatures in the wild.

I’m now on my own private tour of Rwanda; it’s just the guide and me.

Fri, 18 May 2012 00:00:08 -1000

I’m now on my own private tour of Rwanda; it’s just the guide and me.

Wechiau Community Hippo Sanctuary

Thu, 17 May 2012 00:00:08 -1000

The next stop on our itinerary was Wechiau Community Hippo Sanctuary, an ecotourism project in the upper western region of Ghana. The Lonely Planet guide describes it as the most underrated tourist attraction in Ghana. The problem with Wechiau is that it’s difficult to visit. By car it’s almost three hours from Mole, and public transport is limited to tro-tros only, so only the stubbornly determined make the trip. Which is a shame, because Wechiau is awesome.

When we arrived at the sanctuary, we were blown away by the hospitality of the staff. On arrival, they found us a nice comfortable seat and then they presented us with a menu of activities that we could do at the sanctuary. Obviously they had hippopotamus safaris, but they also had a plethora of community visits. We ended up selecting the river water safari and a village walk.

We had a short 10 minute drive out to our accommodation. The lodging is basic, but that’s part of the charm; no electricity, mosquito nets abound, drop toilets and bucket showers. Normally I’d squirm at this kind of thing, but I was so excited to be visiting the sanctuary that I didn’t really mind. They set your expectations well in advance.

Once we dropped our bags off, we headed straight out to our hippo safari. The three of us, a couple from Holland, and two guides piled into a rather rickety looking canoe and headed off looking for hippos. We didn’t have to go far; a group of 13 had formed less than 200 metres from the landing, and so we paddled straight over and started taking photos. Unfortunately there isn’t much of a hippo you can see during the day, but we stayed out there for 40 minutes. It was excruciatingly hot on the water, but I’m glad we got to see such a large group. Back at the camp, I spoke to someone that spent three hours on the safari and could only find one.

After the safari, we visited a local village. The money from the tours goes directly back into the villages; the sanctuary just recently installed two water pumps at each of the 17 villages within the sanctuary.

Back at the lodging, we were supplied with a boiling pot of water to cook our two minute noodles, and after a couple of glasses of wine we retired for the night. The girls decided that they wanted to sleep on the roof, whereas conservative me decided it was better to stay inside. Inside was the better choice, as at 1:00pm it started to rain heavily, and everyone had to come back in.

I could have spent days at the sanctuary, there was that much to see, and I’d highly recommend it to anyone who finds themselves in Ghana. Unfortunately, due to the limited amount of available time, we had to leave early the next morning to head back to Kumasi.

Mole National Park

Wed, 16 May 2012 00:00:08 -1000

Mole National Park is the largest national park in Ghana, located in the North West corner of the country. It’s a bone crushing 670 kilometers from Accra, and the trip generally takes 11 hours. We ended up stopping in Techiman overnight, because the roads are just so treacherous; one minute you’re on a dual carriage way, and the next there are potholes the size of a VW Beetle.

After having some issues with booking the accommodation, we finally were able to book a room within the motel in the middle of the park. It’s a run-of-the-mill motel, apart from the fact it’s in the middle of a national park and the animals are allowed to walk through it. When we were checking in, there was a baboon running along the roof. Once the room was organized, we headed out for our first walking safari.

The Lonely Planet guide describes Mole as being the cheapest safari in Africa. The price for the guide is 3 cedi ($1.6 AUD) an hour, for a minimum of two hours. So we decide to start with a walking safari. We were assigned a guide, Christopher, and with a gun slung across his back, we head out into the park.

I’ve effectively done no exercise for 4 weeks, and walking through the searing Ghanian sun is difficult enough as it is, let alone having to walk through dense savannah scrub. The majority of the first walk was spent seeing glimpses of the backsides of bushbuck through thick vegetation. So we headed back to the motel, where we ordered dinner (I had two beef kebabs and yam chips) and had a few drinks by the pool before it rained us out.

Walk number two was at 7:00am the next morning. Thankfully the terrain was a little more forgiving, consisting primarily of grassland. We did get to see some cute warthog families, and monkeys. It was also refreshing to see the front of a bushbuck, and their cousin, the waterbuck.

I had resigned myself to the fact that we’d never get to see an elephant. We were told that they frequently can be seen drinking from the watering hole near the motel, or indeed the pool, but the “lower” temperature and rain over the two days had kept them away. We spent four hours optimistically watching the waterhole, and then due to a misunderstanding with our driver, we had to rent a vehicle for our driving safari, which we luckily shared with two American girls from Minnesota. It cost 45 cedi ($27 AUD), an hour, for the car, which is outrageously expensive by Ghanian standards.

So sitting on the roof of the car, we slowly made our way through the park. When we saw an animal, we would bang on the roof and the guide would stop. We opted for three hours of driving, and after almost two and a half hours we hadn’t seen a lot. The other safari car that we passed hadn’t seen any elephants, so it was not looking good. As we slowly headed back to the motel, we caught a glimpse of one in the distance, and so we quickly stopped the car and proceeded on foot. It was so exhilarating following the elephants through the scrub, and we eventually got within 20 meters of a male. All happy, we headed back to the motel.

Mole National Park was an amazing experience, and I’m glad we made the trek up here. Next stop, Wechiau Hippo Sanctuary.

View from Mole Motel

Wed, 16 May 2012 00:00:08 -1000

View from Mole Motel

Final presentation

Sat, 12 May 2012 00:00:08 -1000

Our four weeks of work ended on Friday with presentations to the various stakeholders. Adaeze delivered a stirring 30 minute presentation to the crowd, and then we spent the next 15 minutes answering questions as a group. The other two groups then delivered their presentations, and then we spent the rest of the afternoon being filmed individually and in groups. I don’t think the video is going to go viral, but it’ll hopefully provide an insight into the program and the work we’ve been doing.

All of us made it out for dinner as a group, and then it was back to the hotel for some dessert and photo sharing. I currently have over 60 gigabytes of photos, not including my own.

The departure times on Saturday varied, as people flew back to their home countries. Our little trio, of Petra, Piper and I were actually the first to depart, with a grueling 7 hour driver ahead of us to Techiman. It was a sad leaving so many new friends behind, but there were numerous promises of future visits.

Off to Mole!

Final presentation

Sat, 12 May 2012 00:00:07 -1000

Mövenpick

Thu, 10 May 2012 00:00:06 -1000

Mövenpick

Review of Mövenpick, Accra

Clicking Send

Thu, 10 May 2012 00:00:05 -1000

The last week of work has been hectic. We’ve been attending last minute meetings, sending out a constant flow of drafts, and running around Accra picking up gifts for loved ones back home. I’m now the proud owner of a small box of delicious Ghanian dark chocolate, and probably one too many bracelets. Unfortunately the school poster sized map of Accra that I bought for 2 cedi ($1 AUD) from the roadside has been lost in the turmoil.

We had an opportunity this week to check out the IBM office in Accra, and I managed to snap a photo of this:

We just clicked send on our final report, and all that is left now is the presentation tomorrow morning. Good times.

My standard lunch: chicken & rice (Jolloff)

Wed, 09 May 2012 00:00:05 -1000

My standard lunch: chicken & rice (Jolloff)

Jamestown & Bojo Beach, Weekend #3

Wed, 09 May 2012 00:00:03 -1000

Not content with being run off our feet on Saturday, we made Sunday equally as busy. The first stop was Jamestown, the historic quarter in Accra, around James Fort. The area isn’t as developed as some of the other parts of Accra, and we were luckily able to find a guide to show us around (/he just jumped in the bus). We went down to the fish market area, where the locals weren’t too keen on being photographed by a bunch of tourists, but a couple of words and I suspect some cedi later from the guide, won them over.

The next stop was the lighthouse, which had incredible views of the old Accra harbour and the city. I loved the fact that we were able to walk up it. No occupational health and safety when you’re going around the twist.

After we’d seen enough of Jamestown, we headed out to Independence Square. We drive past it every day to the office, but we hadn’t had the opportunity to check it out. We also got to snap some photos of the creatively named Independence Arch.

Next to the Independence Arch is the Accra Sports stadium. After a rather confusing conversation with the guards at the entrance, they gave us a quick guided tour of the stadium.

The crown jewel of the weekend was without a doubt the trip to Bojo beach. About 40 minutes out of town, Bojo beach is an oasis of calm in comparison to Labadi Pleasure Beach. The beach is actually located on a sand bar, and you have to get a canoe across to it. Once you’re there, you can enjoy the clean water, deck chairs, and the obligatory bar. We ordered some lunch at the restaurant, and then went across to the beach for a leisurely couple of hours of swimming.

It’s interesting to note that Ghanians don’t really swim. A lot of the people just mill around in the shallows, shrieking at the sight of waves higher than their waists. I’m not a particularly capable swimmer, but I was complimented a couple of times on my freestyle prowess. Maybe I should move to Ghana and join the swim team.

One of the local restaurants near the office, Country Kitchen, had a special on grass cutter, also…

Tue, 08 May 2012 00:00:03 -1000

One of the local restaurants near the office, Country Kitchen, had a special on grass cutter, also known as the greater cane rat. It came in a peanut soup, with a conglomerated ball of fufu.

Dodi Princess Burnt to Ashes. We were on the last cruise

Mon, 07 May 2012 00:00:03 -1000

Dodi Princess Burnt to Ashes. We were on the last cruise.

Makola Market & Aburi Botanical Gardens, Weekend #3

Sun, 06 May 2012 00:00:03 -1000

Our last weekend in Ghana as a group was going to be hectic. We’d forgone the tour to see the monkeys, instead choosing to spend the time in and around Accra. We hadn’t really had the opportunity to check off all the touristy things to see in Accra.

Starting at 9:00, we headed straight to Makola Market. Makola is the largest market in Ghana, and to say that it’s not enormous would be an understatement. It’s a sprawling labyrinth of street level stalls mixed with three or four story buildings containing even more stalls. We had been allocated an hour to walk around, but some of the group found it a little overwhelming. I loved it. Unlike some of the other destinations in Accra, we weren’t being harassed to buy, so we were free to walk around. The fresh food was particularly interesting, with pigs feet and snails the size of a man’s fist.

Walking around at street level was hectic, with women carrying baskets on their heads, vying for the same roadway as large delivery trucks and other pedestrians. It was busy, crazy, safe fun. Definitely worth a look.

We’ll contrast this to the Arts Centre, which is where they sell souvenirs to tourists. Vendors already start heckling you before the bus has even pulled to a stop. There is some great stuff to buy inside, and some people in the group love the bargaining process, but I just feel rushed and annoyed that I can’t just browse around in peace. Brother, would you like to see my paintings?

Due to the time constraints, we’ve been skipping either lunch or dinner everyday. Today, lunch was being skipped, so we headed off to the Aburi Botanical Gardens within the Eastern Region of Ghana. I’m not particularly fond of gardens at the best of times, but this one I actually enjoyed. Why’s that? Because after exploring some of the different areas, Dwight, Amy and myself trundled down one of the paths carved out by the locals, and we ended up half lost through this thick forest part of the garden.

Before it started to rain, we bundled into the bus and then headed back into Accra for dinner. We ended up at a place called Buka, which has a range of Ghanaian and Nigerian dishes. I had mixed feelings about my goat kontonmire stew. More on that later.

View from the mountains in the Eastern Region, Ghana

Sat, 05 May 2012 00:00:03 -1000

Eating Indian In

Fri, 04 May 2012 00:00:03 -1000

After three straight weeks of eating out, restaurant fatigue is setting in. Although the food we’ve been eating has been generally on the good side (I’m looking at you, Kakum National Park Lasagna), it’s just been too much. It’s been particularly hard on the vegetarian members of the group, who are often only left with rice and cabbage. Even the description of the meal, rice and cabbage, sounds particularly unappetizing.

Luckily, one of the members of our group, Andal, is particularly gifted in the kitchen. She somehow managed to get the hotel staff to let us use the kitchen. So on Monday night, she came home from work early, and cooked up a venerable feast of Indian vegetarian dishes. Not a dressed chicken in sight.

The feast consisted of roti, dhal, raita, cauliflower curry, potato curry, and a sago kheer. We haven’t eaten this well since we arrived, and I loved all of it.

If the first dessert wasn’t enough, we then got treated to some Irish coffee in the lounge.

"Water is Medicine" campaign at the Ministry of Health

Thu, 03 May 2012 00:00:03 -1000

“Water is Medicine” campaign at the Ministry of Health

Banana bread and a “puff puff”. The perfect Ghanaian office morning tea.

Wed, 02 May 2012 00:00:03 -1000

Banana bread and a “puff puff”. The perfect Ghanaian office morning tea.

What all the cool kids are currently listening to in Ghana.

Tue, 01 May 2012 00:00:03 -1000

What all the cool kids are currently listening to in Ghana.

- P Square Ft Akon & May D — Chop My Money

(Source: https://www.youtube.com/)

SOS Children’s Village, Weekend #2

Tue, 01 May 2012 00:00:02 -1000

After a rather long work week, a nice relaxing weekend would have been lovely. However, not wanting to waste any of our precious time in Ghana, we made this weekend as hectic as the last.

On Saturday, we organized a last minute visit to the SOS Children’s Village in Accra. SOS is an organization that runs orphanages around the world. Instead of “Oliver twist” style dormitories, SOS Villages are arranged into small houses, with each house containing between 8 and 12 kids. Each house then has a house mother, who is responsible for running their particular household and caring for the kids. The idea behind this is that it mimics the family environment as closely as possible. Kids stay in the village until they’re 16 or so, when they’re moved to accommodation nearer to high school.

All twelve of us headed off to the village, and after a quick tour of the primary school, it was time to visit the houses. We entered the first house and greeted the mother. The kids, who were watching TV, looked as equally shocked as we were; the flash photography from some of the others didn’t really help. We introduced ourselves individually, which added to the shyness, and then after some awkward attempts at conversations, we proceeded to the next house.

And then kids started to come up to us. They were incredibly fascinated by the array of cameras we were wielding. From tiny point and shoots, to big SLRs, they loved them all. We watchfully handed them over, and 260 shots later, they filled up my CF Card. The little ones just wanted to push buttons, but the older ones were more interested in how the camera worked.

Once, the cameras had exhausted their appeal, some bright spark thought it would be a good idea for us to have a soccer match in the 32 degree, 90% humidity weather. I’m unfit at the best of times , but running around with a bunch of people on a soccer field was particularly challenging.

Unfortunately, it was soon time to leave, but I could have spent all day there. We dropped off our rather large box of presents, and then headed to the bus. The little 4 year old girl who insisted on holding my hand all the way to the door wasn’t too happy to see us go.

PS: The kids took almost all of these photos.

Lake Volta Cruise, Weekend #2

Tue, 01 May 2012 00:00:01 -1000

Sunday we headed out to Lake Volta, a large man-made dam used to generate all of Ghana’s power supply. Apart from it being a picturesque location for a dam, the main attraction is a boat cruise along the lake to Dodi Island. The cruise takes three hours to reach the island, where you then spend 30 minutes, and then three hours back to the starting location. The boat cruise is the experience, not the island.

So we arrived at 8:00 to ensure that we got tickets for the cruise, which can be very popular. We were told on arrival the boat was leaving today at 10:30, instead of the scheduled 9:30. No problem, we just amused ourselves with the limited selection of souvenir sellers. There’s always a chance to haggle for a bargain.

We finally got to board the Dodi Princess at 10:00, where we found a nice set of tables to enjoy the view from. So we sat back, sipped on our one complimentary drink, and just chatted. Soon enough our buffet lunch was served; we got fried rice, jollof rice, coleslaw, and a choice of chicken or fish. It was a little cold, but I wasn’t expecting much for a boat.

The boat eventually arrived at the island and boy were we in for a shock. The tour company had organized some traditional drummers and dancers on the pier to greet the boat, but after the pier there was a swarm of people begging for money. We spent our 30 minutes walking around the island, getting harassed, and then we went back to the boat for the cruise home.

Funnily enough, I actually enjoyed the boating part of the experience; it was relaxing and an enjoyable way to see a different part of the countryside. If they just improved the Dodi Island experience a bit, it would definitely be a lot better.

We ended up getting dinner at a restaurant a little downstream, but we didn’t manage to get back to the hotel until 9:00pm.

Hour five of our Dodi Island Cruise, Lake Volta.

Sun, 29 Apr 2012 00:00:01 -1000

Hour five of our Dodi Island Cruise, Lake Volta.

What's this all about again?

Fri, 27 Apr 2012 00:00:02 -1000

You’d think that after two weeks in, I would have already explained what I was doing in Ghana. Up until now, I’ve been particularly vague about what I’m doing here, just to be on the safe side. However, after a brief discussion with the in country coordinator today, I can apparently say a little more than just complete silence.

IBM has partnered with CDC Development Solutions, a non-profit, non-government organization which leverages volunteer resources to drive economic growth in emerging markets, and USAID, the US Government’s overseas AID program, to develop the Centre of Excellence for International Corporate Volunteerism (CEICV) . The first project of this new, three way partnership, is the project I’m on in Ghana. My project team is working on developing a roadmap for an IT system, for the newly proposed Supply Chain Management Unit (SCMU), which sits under the Ministry of Health. So there you have it, that’s what I’m doing.

Week #2 has been even more hectic than the first. We had a visit out to the Eastern Region of the country to see some of their various health facilities. The 90km trip took almost 2 hours, down roads ranging from four lane highways to dirt tracks. But once we got out there, the scenery was spectacular. Unfortunately, we were there to work, so the only photo I managed to grab was between meetings

Talk to you soon :)

Someone has reviewed one of my favourite restaurants in Accra ;)

Fri, 27 Apr 2012 00:00:01 -1000

Someone has reviewed one of my favourite restaurants in Accra ;)

ANZAC Day Memorial Service at Christiansborg War Cemetery, Osu, Ghana

Wed, 25 Apr 2012 00:00:01 -1000

ANZAC Day Memorial Service at Christiansborg War Cemetery, Osu, Ghana

Cape Coast, Weekend #1

Tue, 24 Apr 2012 00:00:01 -1000

Week one was a bit of a shock to the system. We’ve been running around all over the place interviewing different stakeholders. Sometimes the instructions provided are like “Go to Building X (which is usually in a different suburb), and ask for Person Y”. Somehow, with the help of our ever patient driver, we haven’t gotten terribly lost as yet. On a more important issue, there aren’t any cafes near the office, so we routinely have to drive to Osu, a suburb that’s only 3 kilometers away but takes 20 minutes to get there in traffic.

For the weekend, the group decided to go visit the Cape Coast, a region, you guessed it, along the coast. It’s famous for its beautiful beaches and its plentiful supply of castles. I love a good castle. The Cape Coast is located a numbing three hours from Accra, but we ended up renting a large mini-bus which had plenty of space. Its off-roading abilities were dubiously tested with the last couple of kilometers to the hotel being on a dirt ‘road’.

Elmina Castle was our first stop. Initially constructed by the Portuguese to trade for Gold, it was eventually turned into a hub for slave trading. The lovely, well preserved building, lost some of its gloss when the tour guide described all the atrocities that had occurred within its walls.

Secondly, we visited the Cape Coast Castle, which is the better know of the two. Unlike the first castle, we weren’t bombarded with people trying to sell us personalised sea shells, which was a welcome change. This castle was specifically built for trading slaves, and featured some rudimentary drainage within the rooms.

After a sweltering hot visit to the castles, we headed back to the hotel for a well-earned swim at the beach.

The following day, we headed out to the Kakum National Park. It’s main attraction is the Tree Walk, which consists of a number of bridges strung between trees within the forest canopy. At its highest point, it’s over 40 meters above the ground.

(Adaeze & Francis)

The weekend was an amazing experience. I was however, well tired out by the end of it.

Day #1 — Workshop

Sun, 22 Apr 2012 00:00:01 -1000

Although photos of lions, markets and food are fun, this is unfortunately not a holiday. Monday morning we had a full day workshop with a number of the key stakeholders. Presentations, meet & greets, and brainstorming sessions. It was a lot of work.

(Phone Courtesy of Deiva)

We have three project teams, made up of four people, each working on semi-intertwined health related project. We’re working on specific deliverables as part of a larger body of work, so it’ll take us a while to get up to speed with the current status. We have an entire week of meetings ahead of us, as we try and balance out the various client requirements. It’s hard to separate the forest from the trees, when you can’t identify a forest or a tree.

IBM Ranked Second on "100 Best Corporate Citizens" List

Thu, 19 Apr 2012 00:00:01 -1000

IBM Ranked Second on “100 Best Corporate Citizens” List

Welcome to Accra, Boys & Girls

Wed, 18 Apr 2012 00:00:01 -1000

A wall of heat hit me when I got off the plane in Accra at 9:00pm. An angry, humid, treacherous heat, which was hell bent on showing me who was boss. Our driver met us just out of customs, and we were quickly whisked off to the hotel. A good thing too, as I almost fell asleep on the bus; who knew sitting in a plane seat for 6 hours watching ‘How I Met Your Mother’ could be so tiring?

Our first full day in Accra involved a moderate amount of walking and an excessive amount of sweating. I did get to try tiny boned Guineafowl for lunch.

The following day was a little less of a temperature shock, with a visit to the Textiles Market. The market had a good mix of textiles, paintings, carvings and handmade drums. Just like all markets in the developing world, the stallholders vigorously peddled their wears, and I was luckily saved a couple of times by our lovely guide, Francis. When you get the price from the vendor, you divide it by four, and then you start negotiating.

Accra is a beautifully chaotic city on the coastline. The spaghetti roads, the old stone buildings, the MTN booths on every corner. New buildings are going up everywhere; there’s a lot of development, but you can still walk down the street and see somebody carrying a chicken in a plastic bag.

Jozi Layover

Tue, 17 Apr 2012 00:00:01 -1000

Jozi, Jo’burg, Johannesburg, regardless of its name, is Australia’s gateway to Africa. With a relaxing 22 hour layover in my pocket, I had plenty of time to enjoy some of the wonders of the city, in a safe, fun filled way. Luckily, one of the other CSC participants was taking the same afternoon flight to Accra, so we decided to go out and visit a Lion Sanctuary just out of town.

After precariously explaining ourselves at the front desk, we were introduced to a driver, who had been recommended, and we headed out to the sanctuary. The first part of the visit involved a car based safari, where we saw a number of lions, including an apparently rare white lion.

However, the real highlight occurred after the driving safari. The centre had an area where you could play with lion cubs.

If that wasn’t exciting enough, you could also hand feed a pair of giraffes.

Once we’d finished with the lion safari, we headed to a crocodile farm, which also had a disturbing large section of snakes. Not being a snake lover, I refrained from having one coiled around my neck.

I also had the opportunity to hold a baby crocodile, which unfortunately peed down my arm. If you have the opportunity to hold one, don’t squeeze too hard.

After our little bit of wildlife, we headed back to OR Tumbo International Airport for our leisurely six hour flight to Accra.

Pack your bags

Wed, 11 Apr 2012 00:00:01 -1000

Cooped up in the office every day, it’s sometimes hard to feel like you’re part of something bigger. You work with the same people, deal with same problems, send the same emoticons and go to the same $2 coffee joint. When you read interesting stories like this, there is quite a bit of disconnect between that and the java web-service you’ve spent all day working on.

One of the things that I’m most excited about with the Corporate Service Corps program is the ability to meet, and work with IBMers from around the globe. It was one of the primary reasons for me apply for the program. The Ghana assignment team is comprised of Client Executives, Chip Designers, Application Developers, Financial Analysts and Sales People from every business unit and every continent. Oh, and me. There are very few companies in the world that have the ability to offer these kinds of opportunities to employees.

So it’s my last day before heading off to Johannesburg tomorrow morning at an eyeball bleeding 6:30am. I’ll see you on the other side of the ditch.

Off and racing. Next stop, Johannesburg.

Wed, 11 Apr 2012 00:00:00 -1000

Off and racing. Next stop, Johannesburg.

MEL to ACC to KGL to MEL

Fri, 02 Mar 2012 00:00:00 -1000

Booking flights around, to, or even near Africa, from Australia, is hard. It’s not that you can’t book the flights online, it’s that the familiar online tools such as Expedia and Kayak just don’t have the ability to plan complicated route itineraries that don’t involve flying through a hub city like New York.

To show you the joy, I’ve selected three travel dates that roughly coincide with what I’ve booked. To keep it simple, I’ve selected mult-city bookings at all sites, flying Melbourne to Accra (the capital of Ghana), Accra to Kigali (Capital of Rwanda) and finally Kigali to Melbourne.

First up, Expedia.com.au: It suggests flying KLM the entire way, connecting via Kuala Lumpur, and Amsterdam. Even fly from Accra to Kigali, it suggests connecting via Amsterdam. And the return flight is a leisurely 51 hours. Not great.

Secondly, I tried expedia.com. The US based site usually has better deals, and a greater selection of flights. It suggested flying to Accra via Bangkok and Addis Ababa on Thai and Ethiopian Air. This is actually the route the travel agent recommended. However, coming back, it’s gotten a little bit too creative for its own good recommending Addis Ababa, Bangkok, Guangzhou to Melbourne.

Finally a tried Kayak, which decided it was all too complicated and went home.

So what have I ended up doing? I booked QANTAS and South African Air via South Africa, but I had to book the tickets on two different sites, and I have an overnight stay in Johannesburg.

Only 6 weeks to go.

Teaming Over Time Zones

Sun, 05 Feb 2012 00:00:00 -1000

Part of the CSC program involves a weekly dialup meeting with everyone in the team. When the first meeting appeared in my calendar, the start time for the meeting was 12:00am to 1:00am. Surely that’s not the best time for a team meeting? However, when the team is made up of people from Brazil, Canada, China, Germany, India, Ireland, Singapore and the USA, that’s unfortunately the most appropriate time for the majority of people. Thanks to timeanddate.com for curing my time-zone ineptitude.

Photo From Flickr, By Michele Mazzone

So how does a conference call go with 12 people scattered across the planet? Surprisingly well. There are the standard extended semi-awkward pauses when people try not to politely talk over each other, and the occasional call dropout, but apart from that the two calls thus far have gone well — apart from 12:00am being past my bedtime. We’ve been using the Lotus Live meeting service which makes it a little easier to share presentations and screens between participants.

As part of the preparation, we’ve been given a set of tasks to complete every week, with a number of them centring on team work. We’ve developed introductory participant profiles to help introduce ourselves to each other, and collaboratively developed a code of conduct for the team. Interestingly, regardless of our backgrounds, business units or areas of expertise, we all came up with a similar set of suggestions for the code of conduct. Good teams aren’t restricted by time zones.

IBM Corporate Service Corps

Sun, 29 Jan 2012 00:00:00 -1000

The IBM Corporate Service Corps is a selective leadership program available to IBMers with three or more years of service. You’re assigned to a team of approximately 10 colleagues from around the world and given a project to work on in an emerging market. The project involves two months of after hours pre-work and then a month of work within your designated country. Click here if you’d like to know more about the program.

So I’ve been assigned to a team that will be working in Accra, Ghana. What do I know about Ghana? Well apart from the fact it’s in Africa, not a lot; I’m hardly a global citizen. So over the next two months I’ll be learning a lot more about Ghana and the CSC program, and updating this tumblr.