RK4 in Scala

I’ve been looking at Scala recently and I’m impressed. It’s got a nice mix of imperative and functional styles that means that it can be really useful and easy to code in.

As an example I thought I’d knock up an implementation of RK4 in it.

The work-horse of RK4 is implemented as follows:

def rk4Iter(f: (Double, Double) => Double,
            t : Double,
            y : Double,
            h : Double) = {
  val k1 = f(t, y)
  val k2 = f(t + h / 2, y + h * k1 / 2)
  val k3 = f(t + h / 2, y + h * k2 / 2)
  val k4 = f(t + h, y + h * k3)
  y + h * (k1 + k2 * 2 + k3 * 2 + k4) / 6
}

One thing I think we can all agree on is that having functions as first-class objects makes this incredibly succinct and easy to follow. I’m just calling the function f a whole bunch of times. Coming from a Java background this is pleasantly less noisy (compare with f.calculate(t, y) etc).

Another subtlety is the lack of specifying a return type. Scala is smart enough to infer the return type

This method is intended to be called from a loop that performs n steps of size h:

def rk4(f: (Double, Double) => Double,
            t0 : Double,
            y0 : Double,
            n : Int,
            h : Double)
    : Double = {
  n match {
    case 0 => y0
    case _ =>
      val y1 = rk4Iter(f, t0, y0, h)
      rk4(f, t0 + h, y1, n - 1, h)
  }
}

For those coming from a Java background you’ll notice that match looks a lot like switch. You can think of it as switch on steroids (I’ll leave off explaining more as it’s a blog post in its own right!). It is sufficient to say that case _ is similar to default.

In functional programming the preferred way to do things is using recursion rather than iteration (when written properly the compiler can identify tail-recursive functions and make them just as fast as loops). We do this because it makes them easier to mathematically reason about them. Suppose the function works for:

• the 0th case
• the nth case if it works for the n -1th case

then the function works for all positive integers.

A recursive style also reduces the scope for off-by-one errors.

And now for an example that ties it together:

rk4(
  (t, y) => 4 * Math.exp(0.8 * t) - 0.5 * y,
  0, 2, 10, 0.1)

I think anyone coming from a Java background will agree that passing around functions like this is lovely: goodbye interfaces and lots of curly braces, hello Functional Programming.

Launch of Two New Sites

Today I see the launch of two sites of mine that I’ve been working on in my free time over the past month or so. Neither of them are finished and well polished, but, the act of getting them out there will ensure I start taking them more seriously and start getting them used

So, I am proud to present: No Green Sheep and Self-review.com.

No Green Sheep

The idea behind this site is to promote being more green and friendly to our planet – but by understanding the effects of our actions instead of just doing whatever someone tells us.

The inspiration for this was a friend boiling a kettle on a gas hob and telling me it was more eco-friendly than boiling in an electric kettle. I asked “how?” and they couldn’t explain or give any numbers… unsurprisingly, I haven’t changed my kettle boiling habits. If my friend had known a few more details about this then she might well have had a convert out of me!

Self-review.com

When I worked at CHP Consulting I kept a monthly reminder in Outlook telling me to review how I’ve done over the past month. This process of self-reflection allowed me to move forward and improve myself in a positive manner. It also gave me complete ownership of it – it’s far better to set your own goals and achieve them than have someone else do it for you.

After leaving CHP I no longer had Outlook. I also decided I needed something a little more guided – something I could just tick a box to say I’d done what I wanted to do. I also wanted to gain feedback from my friends whenever possible (a self-criticising blog post of mine made me realise I wanted this). So, I made self-review.com. The commenting isn’t yet ready – but I’ll keep you posted!

My first review of myself is available at http://www.self-review.com/reviews/5

Twitter Accounts for Both

If you want to keep up-to-date with both of these projects then they have their own Twitter accounts: selfreview and nogreensheep

Why We Made Our Own Bug Tracker

Part of starting the development of Sonnet Models was choosing a bug tracker. Knowing that reinventing the wheel was a bad thing I decided that we should do it anyway.

Why would I do this? Knowing full-well of the existence and having used numerous bug trackers I decided that they all suffered from one common flaw: bloat.

Bloated with Fluff-Fields

If entering a bug requires more than 20 seconds of entering other fluff (such as priority, origin, area, etc etc) then small bugs will simply not get entered most of the time because it takes too long. It’s hard to admit that people are lazy but I’ve seen it happen too many times, “I’ll just add this bug onto this other issue to avoid trying to remember what to put in field X”.

I also believe that people tend to rush filling in such fluff-fields and the result is that much of the data is junk.

What do we Track?

In our bug tracker we have nine fields per bug.

Three of these are automatically generated and cannot be modified (created date, creator, modified date).

The other six are: type (bug, feature or admin task), summary, details, timebox, status (created, ready for testing, pending release, closed) and assignee. That’s it. I’m even considering binning the type field as we rarely use it (fortunately it defaults to bug so there isn’t an admin overhead here).

We allow commenting – but there is only one field – comment.

Drawback – No Sexy Reporting

Sure, I can’t query the bugs database to see the most common origin of bugs, but, as a start-up we have a very good feel for this and knowing the exact numbers wouldn’t justify the cost in time of collecting this data.

I also feel that the lack of sexy reporting is an advantage – I don’t spend half my afternoon trawling through meaningless reports looking for imaginary trends!

Question for Large Companies

Do you actually get any use from all the random data you collect? Or is it slowing you down? Are bugs slipping the net because people don’t want to have the burden of reporting bugs? (Or the e-mail deluge that normally follows from reporting a bug).

Avoiding the fluff is a hard task and sometimes we should learn to just say “no, we can’t add a field onto our bug reports saying what coffee the reporter was drinking at the time”.

Agile Zen

I’d like to finish this post by mentioning a new tracker: Agile Zen. This is an immensely light weight tracker and if it had commenting… we’d switch over as quickly as possible!

HtmlUnit – How to do a POST

Background

I’ve just started using HtmlUnit (I also use Selenium but find it too slow for 90% of my testing).

HtmlUnit is generally easy to work with. Generally, it doesn’t need documentation as the Javadoc is useful and the methods are obvious. However, this is not the case for POST… so, here is how you do a POST

How to do a POST in HtmlUnit

There really isn’t much to it – you just need to know what code to use:

final WebClient webClient = new WebClient();

// Instead of requesting the page directly we create a WebRequestSettings object
WebRequestSettings requestSettings = new WebRequestSettings(
  new URL("URL GOES HERE"), HttpMethod.POST);

// Then we set the request parameters
requestSettings.setRequestParameters(new ArrayList());
requestSettings.getRequestParameters().add(new NameValuePair("name of value to post", "value"));

// Finally, we can get the page
HtmlPage page = webClient.getPage(requestSettings);

Quite a lot of work for a simple… I imagine it won’t be hard to wrap this up into a neat POST method.

I feel good! (Follow-up on Lessons Learned!)

I feel good. I am more productive and happier in my life.

Why?

I’m working less.

Why?

The conclusion of my previous blog post was that I should try to work less. So, I’ve been forcing myself to ~7-8 hours every day.

Working less but getting more done?

Yes. I am motivated because I have had time to relax and reflect.

Win?

You bet.

How I Have Failed – Lessons From My First Four Months of a Startup

Over the past few days I have taken some time to reflect on the previous few months and think about how I have failed in various ways and also how I have done well

As some of you will know I left my job in February to become the technical director in an early stage startup (actually I do everything technical and also a bit of the business work – that’s the joy of a small team!). Since then we’ve done a lot of great work, but I have also done my fair share of things that should now be thought of as learning opportunities. I’m going to take some time to go over a few of them as I think I’m not alone in making these mistakes!

Disclaimer: This is a bit of a brain dump – mostly self-indulgence for me in actually straightening out my thoughts!

Not Pausing For Reflection

It’s taken me four months to sit back and go “Are we doing this right? How am I behaving? What should I be changing?”. Sure, I’ve paid lip service to doing this but I’ve not properly sat down and given it serious thought. This is far too long to go without doing this – I’m confident I would be guilty of committing fewer of my sins below if I had done this.

So, I’m now going to sit down once a month and have a good think

Not Releasing Quick Enough

The product that we are developing is now in a public beta and we’re about to go for a big launch. We could have launched a few months ago with fewer bells and whistles. This would have given us far more time to listen to what people are saying instead of guessing and making a few mistakes. On the flip side, we could have wasted our opportunity to build momentum as people saw it and thought “this is rubbish”. However, that doesn’t seem to have been a problem for Facebook, MySpace, Twitter, etc… I think that most people these days are quite happy to accept an evolving service – so long as it isn’t breaking. I also think this is a good way to build and sustain hype – people talk about new features / design tweaks.

Not Treating Close Friends with Sufficient Respect

This is a haenous crime and I apologise to everyone who has been on the receiving end of this.

As my startup is basically my life I’ve told my closest friends quite a lot about it. This means that when they later talk to me I forget that they aren’t as absorbed as I am and don’t know my reasoning behind things when I shoot down a suggestion from them. A typical conversation might go:

Them: I think you should do X

Me: That won’t work. The demand isn’t there for it.

And I’ll try to end the conversation there. Bad idea. Often the idea is a great idea there is just some nuance of the business that makes it inapplicable in our case. I should thank them for their idea and explain to them my reasoning for any bold statement about the idea.

Optimising Early

Guilty. I did some scaling testing and scalability was OK… but I wanted more bang for our buck. This is the sort of thing that (provided you know how you’re going to do it) can be done later when you’ve actually proven the business is meeting a demand.

Working Too Much

It’s very easy to get absorbed into your very own startup. So easy that I have ended up spending a lot of time either working on it or doing things very closely related. The result: friendships suffer, my concentration suffers and my ability to make good decisions becomes less… good. I’m now setting a rule of no work after 8pm and preferably no work after 6pm

Not Networking Enough

I’ve blamed this on moving away from London and the time it takes to commute there and back. In truth, I’m lazy and I’ve been working too much. If I’m sensible about how I work then I can easily get myself to some good events, have a good time and meet some great people!

I’m going to start making myself get to at least one event a month… am open to fun suggestions in London!

Allowed Things to Drag

Certain business decisions we’ve made have been allowed to drag on for far too long. We’ve tried to be clever about it and get the best timing, make the best decision etc etc… but I think this has been us covering up for indecision. If we don’t know how to make a decision we should be thinking about what we need to know to make that decision.

I’m going to start forcing decisions – or forcing gathering the knowledge we need – or developing an action plan for getting somewhere with the decision

Not Blogging Enough

This is a similar point to the reflection point. The act of blogging is a great way for me to get my thoughts out and clear my mind. I can then focus far better afterwards!

I’m going to try to blog once a week

My Focus For Personal Development

I know it’s hard to improve several things at once… so, I’m going to focus on not working too hard and hope the rest falls into place. In a months time we’ll see where I have got!

One last thing… The Good?

To summarise the good: we’ve developed a really neat looking product that we’re proud of it. I’m proud of it I’ll share it with you all soon! (I realise I’m continuing to violate “not releasing quick enough” but there’s only a few days left till we launch so might as well hold on!)

Windows 7 (World) vs Ubuntu 9.04 Jaunty (World) for a developer

Background

This initially started of as a blog posting about how wonderful Ubuntu is for a developer and the various pitfalls I found. However, since installing Ubuntu and initially being wowed I have gone back to Windows. So, I thought I’d offer up my thoughts as to why I made this decision.

For the record: I still use Ubuntu on my laptop because some of my problems with Ubuntu aren’t evident on my laptop. I do also have Windows on dual-boot for the times I need it.

This is not Windows vs Ubuntu

The (world) in the title is a hint. I am comparing my world when running Windows  against my world when running Ubuntu. This means that I am looking at not just the operating system but how I can go about doing everything that I do (mostly development work with a bit of gaming thrown in)

Ubuntu – The Good

So, what makes the Ubuntu experience better than Windows?

• Out the box support for multiple desktops
  • That said, the apps for Windows that do multiple desktops are much more stable than a few years ago
• Customisation of… everything
  • For example, on my laptop I have compiz set up to remove title bars from maximised windows – screen estate is valuable on a laptop
• Automatic updates for everything
  • Nearly everything I install I use apt for. This then means it gets auto-updated with new features / bug fixes
  • Nearly everything in Windows has its own update manager… and I kill most of them as I don’t want them running all the time

Ubuntu being free isn’t such a winner for me – it only adds to the experience when you first install it and you don’t continuously feel the benefits

Ubuntu – The Bad

And… what makes the Ubuntu experience worse?

• Initial fonts (in my opinion) are horrible
  • This isn’t really Ubuntu’s fault. Many websites are designed using Windows fonts and due to licencing the fonts can’t be installed by default
  • Solution: install windows font pack and various other bits and pieces
• GIMP. I’m not a designer. I want to take screenshots, resize them and draw cirlces / boxes around things. I don’t want to learn anything new to do this
  • GIMP by default is the equivalent of using Adobe Photoshop by default – it’s a powerful tool but not the right tool for my simple tasks
  • In Windows, Paint.net is exactly the right tool for 99% of the image editing I do, and, I haven’t had to spend time learning it
• Dual-screen support
  • This was a huge chore to get working right in Ubuntu – it defaulted my left monitor as my primary monitor and setting it otherwise was non-trivial
• User interface sluggish
  • After doing some reading it seems my ATI card is to blame
• ATI drivers
  • Couldn’t get these working… but installing/uninstalling them was a lot of grief – if ATI want me to buy another ATI card then they need to support Ubuntu better (I hear NVidia has far better support)
• SQLYog doesn’t work as nicely in Wine as it does in Windows and I couldn’t find a linux alternative
• No equivalent to TortoiseSVN
• Customisation of… everything
  • A blessing and a curse. I spent far more time than is healthy playing around with things
• Gaming
  • I play Team Fortress 2 to unwind – it’s not as simple to get started as it is in Windows

Conclusion

So, what made me can the Ubuntu experiment? Dual screens and the sluggish reponse. These are the only things that affected me on a continual basis, everything else I could deal with.

In defence of Ubuntu: most of the problems are not the problem of the OS but a lack of support from other people and a lack of linux equivalents to my favourite Windows programs.

I hope this is of some use to anyone with a similar lifestyle when deciding what OS to use As always, comments are welcome!

Redis vs MySQL vs Tokyo Tyrant (on EC2)

Update (28th April 2009) – Added Tokyo Tyrant

What is Redis?

Briefly, Redis is a key-value store that stores the values to disk periodically instead of after every set.

What is Tokyo Tyrant?

Tokyo Tyrant is similar to Redis – a key-value store.

Aim

This is a brief comparison of the performance of Redis and MySQL. It is not very thorough and is only to be used as an indication as to whether Redis may be worth your further investigation.

After a brief search I failed to find any performance comparisons of Redis and MySQL to determine if it was even worth my time looking at Redis. I appreciate they are very different beasts but in our database we have a lot of data that would fit very nicely into a key-value store.

So, the question I am trying to answer is “roughly how much performance can we gain from using Redis for storing simple data instead of MySQL?”. Then it is up to you to look at all the differences and decide whether it is worth pursuing a switch to Redis.

Note: I will be looking at other key-value stores in the near future

The Test

The two tests performed are:

• Using 2 threads: set X items with the keys/values being consecutive numbers
• Using 2 threads: randomly get these items Y times

Setup

1. EC2 small instance.
2. Default install of MySQL with a single InnoDB table for storing keys and values.
3. Default make of Redis.
4. Small Java benchmark programs sitting on the same box

Source Code

The numbers in these files are not the exact numbers I used (apart from using two client threads in both tests)

1. Redis benchmark: http://pastebin.com/f5adb41b0
2. MySQL benchmark: http://pastebin.com/m52fd35d1

Results

Mysql

3,030 sets/second.

4,670 gets/second.

Redis (In-memory only)

11,200 sets/second. (3.7x MySQL)

9,840 gets/second. (2.1x MySQL)

Tokyo Tyrant (Writing to disk)

9,030 sets/second. (3.0x MySQL)

9,250 gets/second. (2.0x MySQL)

Thoughts

Key-value stores are generally going to be faster than MySQL. They (generally) provide less functionality in the way of querying arbitrary columns and ACID compliance. This translates into faster performance on gets/sets.

Once again, it is worth pointing out that this is a very simple benchmark. My suspicion is that when you have lots of cross updates going on then Redis/Tyrant will have a greater performance boost over MySQL – but at the cost of transactionality.

My suspicion is that as the technology matures we are going to start seeing a big shift towards using key-value stores as the first port of call for data storage for websites. I expect it won’t be long before we start seeing hosts offering LAKP (Linux, Apache, a key-value store, PHP) as well as the more traditional LAMP offerings.

I would love to hear from anyone with real world experience and numbers of switching over to Redis… until I have some of my own to share anyway

C3P0: Tried to check-in a foreign resource!

Earlier today I encountered the following stack trace whilst using C3P0 and Hibernate:

com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool - An Exception occurred while trying to check a PooledConection into a ResourcePool.
com.mchange.v2.resourcepool.ResourcePoolException: ResourcePool [BROKEN!]: Tried to check-in a foreign resource!
	at com.mchange.v2.resourcepool.BasicResourcePool.checkinResource(BasicResourcePool.java:657)
	at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$ConnectionEventListenerImpl.doCheckinResource(C3P0PooledConnectionPool.java:636)
	at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$ConnectionEventListenerImpl.connectionClosed(C3P0PooledConnectionPool.java:630)
	at com.mchange.v2.c3p0.util.ConnectionEventSupport.fireConnectionClosed(ConnectionEventSupport.java:55)
	at com.mchange.v2.c3p0.impl.NewPooledConnection.fireConnectionClosed(NewPooledConnection.java:510)
	at com.mchange.v2.c3p0.impl.NewPooledConnection.markClosedProxyConnection(NewPooledConnection.java:381)
	at com.mchange.v2.c3p0.impl.NewProxyConnection.close(NewProxyConnection.java:1246)
	at org.hibernate.connection.C3P0ConnectionProvider.closeConnection(C3P0ConnectionProvider.java:92)
	at org.hibernate.jdbc.ConnectionManager.closeConnection(ConnectionManager.java:474)
	at org.hibernate.jdbc.ConnectionManager.aggressiveRelease(ConnectionManager.java:429)
	at org.hibernate.jdbc.ConnectionManager.afterTransaction(ConnectionManager.java:316)
	at org.hibernate.jdbc.JDBCContext.afterNontransactionalQuery(JDBCContext.java:268)
	at org.hibernate.impl.SessionImpl.afterOperation(SessionImpl.java:444)
	at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1153)
	at org.hibernate.impl.QueryImpl.list(QueryImpl.java:102)

Apologies for the scroll bars… but hopefully this is indexed neatly by Google to help others!

After a bit of investigation it turned out this was due to a multi-threading issue and the solution can be summarised quite neatly:

never shut down Hibernate before all threads have finished what they’re doing (if they are using Hibernate)

Otherwise, you may get the above exception (or an equally unhelpful exception).

How I Test and Tweak CSS for Cross-Browser Compatibility

Motivation

Why bother supporting more than one browser? For anyone wanting to hit a large proportion of the population then supporting Firefox 2, Firefox 3, Internet Explorer 6 and Internet Explorer 7 are necessary to satisfy 89% of the market (browser stats). It may also be desirable to support Chrome and Safari to get to 96%.

That is a lot of platforms – all with their own behaviours and interpretations of specifications. I’ll ignore arguments about upgrading browsers / converging on one/two browsers as that is a big topic that is regularly argued about the world over.

So, here is how I cope with the joy of cross-browser compatibility

Testing

Before we can fix something we need to know it is broken. The approach I take is to try out the web site on all the browsers I want the site to work on – unfortunately, it’s not that simple to have all possible browsers on one machine.

Firefox 3 and Internet Explorer 7

These two are easy – I have them installed all the time on my development machine. They co-exist quite peacefully and give me no grief.

Firefox 2

Firefox 2 can be completely standalone as a portable app. Stick it in a folder and run it, at the same time as FF3. Here’s how

Internet Explorer 6

Rumour has it that you can install Internet Explorer 6 as a standalone application. I’ve had no luck with this approach. My approach is to use a virtual PC. Microsoft have kindly made Virtual PC 2007 available for free here. They also provide images for the common setups here.

Safari

This is available from Apple as a download for the PC, here.

Tweaking for Compatibility

So, we’ve now identified a problem with our site in one of the browsers. Searching the internet will usually reveal the necessary tweak for that browser. There are also countless tricks out there for making our CSS only appear to a particular browser. Such as:


/* Hide from IE-Mac \*/
#header {margin-bottom:3em}
#footer {margin-top:1.5em}
/* End hide */


I generally don’t like these approaches as they obfuscate our CSS to some extent, this then leads to mistakes. So, what do I prefer?

Browser specific CSS in browser specific files. For any of my sites my stylesheets will generally look like:

• default.css – included by all browsers for the core style
• ie6.css – included only by Internet Explorer 6
• ie7.css – included only by Internet Explorer 7
• ff2.css – included only by Firefox 2
• Firefox 3 is omitted as I develop in Firefox 3 so that all stays in default.css

The important thing here is that the browser specific files are included after the default. This allows me to specify that a div shouldn’t be a float but instead should use relative positioning or whatever specific tweak is needed.

I feel that this yields much clearer CSS than the hack approach and is hence more maintainable. This also allows us to completely overhaul a design for a particular browser if it is giving us substantial grief. The disadvantages to this approach are that it increases the number of files to maintain.

My personal view is that I would rather have 4 clear and easy to maintain files rather than 1 hard to maintain file.

Implementing this is fairly easy in most systems. In PHP we can use $_SERVER['HTTP_USER_AGENT'] to get to it. We can even use Javascript / AJAX to do this.

But, isn’t this unreliable? Couldn’t a user fake this? (Or, a firewall). Sure. However, we’re trying to be pragmatic here – we can’t get a complex page to work 100% reliably on all browsers without doing CSS tricks so we have to do something that is maintainable.

If we keep as much as possible in the core stylesheet then the user shouldn’t have a terrible experience if their browser is mis-reported and so we shouldn’t be losing too many visitors to bad browser detection. We can even go one step further and ensure that the core style sheet gives a good (not perfect) view of the page in every browser without additional tweaking… then layer the tweaks on top!

Disclaimer

In the above I have outlined a method that may or may not be appropriate for your situation. If you have a little site with one page, a simple layout but it doesn’t work in all browsers then you’re probably better off using hacks. If you have a big site with lots of pages and a complex layout then you may want to consider layering your stylesheets.

As with all software development – use your brain. Read around. Make the decisions that are right for your project – never use a fork to eat soup.