Greenlit!

Exo-Miner

Not so much of a programming entry, more that I'm rather chuffed that we've finally decided on a name for our game. We've been using the working of 'SolarEscape' for ages but when you google that, you get a whole bunch of tanning salons pop up and anyway, the game's scope now goes way beyond just escaping the sun.
  Anyway, SolarEscape is now Exo-Miner which neatly describes what you do and rolls off the tongue quite nicely. In the game you'll explore procedurally generated solar systems, prospect asteroids and defend your patch against rival factions.
  The title screen image was created using a combination of Clayworks for the modelling, the game engine itself for the procedural nebula in the background and blender+cycles for rendering and composition.

More on this soon!

Visual performance analysis

Eventually, every game needs one of these:

In order to get the game running at at least 30fps, it's important to know where time is being spent and where to focus optimization efforts.  Early on in the development of Solar Escape, I implemented a timer system to log how long various tasks take - it's a simple piece of code that I think I must have written dozens of times over the years.
  The coloured bars each represent an event in rendering the frame. These can be nested so more general events (update, rendering, etc) are shown higher up and more fine grain events (drawing an individual object, for example) are show further down.  When hovering over the mouse over a bar, the name and duration of the events is displayed and the event itself is high lighted. The yellow bars represent time blocks of 10 milliseconds. The green bars represent targets for 30fps and 60fps (if our general's ribbons are to left of both green bars, we're in a 60fps happy place). Solar Escape is a fast moving game and a nice smooth framerate makes all the difference when playing.
  This is all pretty straight forward but it is an invaluable tool and I wish I'd implemented the visualiser earlier on in the project - the real time feedback and whole-frame visualization allows us to rapidly determine where we're spending most of our time each frame and also to spot, at the time of the event, when issues arise. Debugging performance issues can be a pain partly because you can't debug these issues as you would other errors - setting a break point on a potential issue distorts timing calculations and doesn't often yield any useful answers when debugging performance issues.
  Also, printing out timing information in the console is often not very useful unless that data is formatted well and even then, it quickly becomes overwhelming. Furthermore, outputting a lot of text to the console tends to be more expensive than drawing the visualization seen above. I fill a vertex buffer each frame containing the vertex information for around 4000 events - that takes much less time than sending all that information textually to the console and it's easier to read, too. One issue that the performance timer revealed was that rendering the UI was as expensive as rendering all of the per-pixel shaded, post processed in game graphics. This was due to older code not feeding the GPU correctly and failing to batch data - if you rendered each rectangle individually instead of filling a large vertex buffer, you'd spend all of your time rendering the performance visualization which would kind of ruin the exercise.
 From the above screen shot, there are a few areas that need attention.  The large blocks of cyan and red represent Box2D's physics processing. It seems some physics objects are not being disabled when they leave out of the visible play zone and we're spending a lot of time there. The white and green blocks to the right represent older code performing scenegraph and partitioning recalculation - both of which have a lot of obvious areas in need of optimization.

  The first mistake I usually re-make each time I implement one of these is not using a high resolution timer. Fortunately both windows and Unix based systems support high resolution timers in the order of microseconds or millionths of a second (or even nano seconds). The standard resolution timer on windows only returns values in milliseconds (thousandths of a second). If you're implementing your own visualiser, make sure you use 'QueryPerformanceCounter()' instead of 'TimeGetTime()' on windows. Unixes can use the 'clock_gettime()'  function/

The most useful optimization that these kind of tools provide is of development time - making games is hellishly time consuming!

We should teach the beauty of Mathematics

Have a look at this link for Michael Gove's vision for mathematics education

Gove

In school, we learned how to multiply fractions. I can distinctly remember turning to my friend and saying 'Well, that's it then. We've learned maths. We know addition, subtraction, division, multiplication and fractions - what else is there?'. To my shame, I genuinely thought that I'd exhausted the subject. How could I be so incurious and arrogant to think that I knew everything about maths? Thirty years of learning later, I've mostly worked out how little I know. I'm not even sure the map of my ignorance is anywhere near complete.

   One thing I have learned is that Maths is far from the dreary subject we had to put with in school - it's beautiful, complex, universal, satisfying and the basis of (or at least useful to) just about every interesting vocation and to our very understanding of the universe. If we ever meet another space faring species, they'd be unlikely to speak any natural human language. I think that we would be able to communicate using mathematics, however: You'd be unlikely to get in to space without it.

   So how is it that our education secretary, Michael Gove, is harking back to look-up-tables and learn-by-rote. Times tables are heuristics, a small set of short cuts that do not scale well. Try asking a kid raised on times tables what 202 x 32 is - they can't, unless they have another method. As a kid, I worked out that you could split any complex multiplication in to simpler ones, honing in on the target number quite quickly. At the very least, you can get the magnitude quickly, even if you're off by a few smaller digits.

   I don't want to state the obvious but you only really need to know short cuts for multiplying by 2, 3 and 5, addition and that adding a 0 to the end of a number multiplies by 10.  You can composite answers using factoring and far less rote memorization - learning how to play with expressing a question in different ways in fundamental to learning to use mathematics creatively so that's a more important lesson than a head full of dumb tables.

202 * 32 = (200 * 32) + (2 * 32) 
200 * 32 = 32 * 100 * 2 = 3200 * 2 = 6400
2 * 32   = 64 
202 * 32 = 6400 + 64 = 6464 

Or a more 'time table' example:

7 * 9 = (7 * 10) - 7 = 70 - 7 = 63

 We teach kids times tables up to 12 and then stop - it's arbitrary and largely useless for anything than counting change whilst shopping. I'd rather we gave our kids the tools to aspire to something more fulfilling and worthwhile than merely the efficient purchase of consumer goods.

   There's nothing wrong with a look-up-table for a given task: Most programmers know powers of 2 off by heart (2, 4, 8, 16, 32 etc.) because that sequence is useful to our profession but more importantly, we appreciate why. The method and rational behind it is much more important than the mindless repetition of numbers.

landscape entirely generated by mathematics (c) Inigo Quilez

  My early experience of mathematics in school, as with many kids, was dull.  I did have a brief introduction to SMP, which was the most positive school mathematics experience I had and the polar opposite of Gove's Victorian rote learning. However, they they phased that out rather quickly.
Everything changed when I started teaching myself programming, particularly graphics programming and discovered how useful, how beautiful maths could be.

  If you have a fast computer and a modern web browser, check out ShaderToy. This website contains beautiful animations entirely generated by mathematics - there's no art or modelling software used to create these other than a text editor and the author's vision expressed through mathematics and GLSL shader code, which is all available for you to tinker with directly on the website.
  That's what schools should show kids - just look at this, this is what maths and programming can create: Stunning, beautiful flights of the imagination expressed in a few pages of code.

  We should be showing kids that their computer games are made from maths, how that third person camera using quaternions to interpolate orientation, how the light reflecting from that burnished metal is calculated using vectors and matrices and fresnel equations, how the A.I. is weighing up probabilities before deciding how to counter the player's move. It's all maths - times tables are a dull footnote. As interesting to maths as a pencil sharpener is to art. 

  And yet, they seem central to Gove's vision for the next generation's education in the UK. This is not the vector along which we should be heading. Gove has also been accused of side-lining arts and music in favour of his stunted vision of science - again, that is wrong. Teach the art and beauty in Maths and Science, teach the Science in art too - as ShaderToy shows, they're often the same thing.

  I have to conclude that either our current administration is either like my 9 year old self - ignorant to the true depth and beauty of mathematics as demonstrated by the fact that they place inordinate weight on archaic teaching methods or else they know exactly what they're doing and want to create a population as ignorant as the child I once was.

When I'm trying to code something I typically start with one problem and end up with fifteen, one of which will involve regular expressions. Today, I wanted to revamp my old Clayworks website as it's a stuck-in-1999 embarrassment (not quite geocities bad but bad enough).
  I'm using sublime text for my HTML editing but I found that I needed to re-format my old code and sublime-text doesn't have that functionality built in.
  Sublime-text does have rather nice python integration for easy plugin development so I did a little search, found this chap's website (http://www.bergspot.com/blog/2012/05/formatting-xml-in-sublime-text-2-xmllint/) and adapted his code, which makes use of xmllint (a unix tool but here's a windows version: http://code.google.com/p/xmllint/).
  My version doesn't need the text to be selected (if text is selected, only that part will be beautified) and will shows a pop-up error message and also takes you to the error line.

Anyhow, here's the code:

import sublime, sublime_plugin, subprocess

def find_lint_error_line(errString):    
  startpos = errString.find("-:")
  endpos = -1
  err_line = -1;
  if startpos != -1:
    startpos += 2
    endpos = errString.find(":", startpos)
    if endpos != -1:
      numstr = errString[startpos:endpos]        
      if numstr.isdigit():
        err_line = int(numstr)        
  return err_line
 
 
class TidyXmlLintCommand(sublime_plugin.TextCommand):
  def run(self, edit):
    command = "XMLLINT_INDENT='\t' xmllint --format --encode utf-8 -"
    self.view.set_status('self', "")
    #xmllint.view.set_status('Hey there Tim', "Hey hey hey")
    # help from http://www.sublimetext.com/forum/viewtopic.php?f=2&p=12451
    xmlRegion = sublime.Region(0, self.view.size())
    p = subprocess.Popen(command, bufsize=-1, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE, shell=True)

    sel_regions = self.view.sel()
    selection = sel_regions[0]
    sel_region_count = len(selection)
    if sel_region_count == 0:      
      selection = xmlRegion
      
    result, err = p.communicate(self.view.substr(selection).encode('utf-8'))    
    
    if err != "":          
      self.view.set_status('xmllint', "xmllint: "+err)
      error_line = find_lint_error_line(err)
      #err.parse()
      #lines[] = parse("line {}", err)
      if error_line != -1:            
        pt = self.view.text_point(error_line, 0)

        self.view.sel().clear()
        self.view.sel().add(sublime.Region(pt))
        
        self.view.show_at_center(pt)
        #self.view.set_viewport_position((0, error_line * 16), True)      
      else:
        sublime.message_dialog('could not find error line')

      sublime.message_dialog('xmllint, error at line: '+ str(error_line) + " \n" + err )

      sublime.set_timeout(self.clear,10000)
    else:
      self.view.replace(edit, self.view.sel()[0], result.decode('utf-8'))
      sublime.set_timeout(self.clear,0)

  def clear(self):
    self.view.erase_status('xmllint')

And if you're wondering how I did that nice syntax colouring on the python code, check this out: SyntaxHighlighter

Further adventures in android NDK

Following some good advice on forums, I've gone and got an NVidia Tegra based tablet (Nexus 7). The reason for doing this was to get access to their android tools, one-stop-shop tools for android development. It'd be nice to say that the installation was entirely painless but, unfortunately, that wasn't quite the case. Partly, this was due to our internet connection dying which, due to this being a massive ~2 GB download, wasn't very nice. Eventually, I did get it to download but there were still some troubles: The kit includes a nice visual studio integration (which is great, that means I have more options) along-side a tailored release of Eclipse.  When trying out their 'hello-world' example, I got various errors that would change each time I downloaded the NDK (which I had to do several times).
  Seeing as my co-worker managed to download and install it without problems last Friday, I think what may have happened (although I have no direct proof of this, take with a pinch of salt) is that the server has been updated in between then and now and therefore, I received a corrupted download. For one thing, his installation includes this directory:  c:\NVPACK\NVNDK , which mine does not. That directory contains another copy of the android SDK and the first time I installed the SDK, the Visual Studio template 'hello world' example project referred to that directory.
  Today, when I cleaned, reinstalled and recreated the template sample project again. This time, it didn't refer to the NVNDK directory but still referred to a tool-chain that didn't exist (4.7.2). Fortunately, you can select a new tool-chain (visual-studio10->project-properties->android directories->Toolchain version) and it all just works. And debugs!
  The next problem I had was on my Nexus 7 - it wan't appearing on my 'device' menu in visual studio or in eclipse. On my old Samsung phone, you had to enable developer mode in the settings but I couldn't find the option on the Nexus 7. After a bit of forum hunting, I found that you have to click seven times on (on the tablet) 'settings->about tablet->build number' in order to enter developer mode. That's pretty cryptic, guys.

Incidentally, if you click many times on 'android version' you get a picture of a jelly bean (on this version of android (code name: jelly-bean), anyway.) Swiping the screen gives a little 'game' where you flick jelly beans around. Doing the same on my Galaxy S II, I get this picture:

Android & gingerbread zombies by Jack Larson
Easter eggs are great.  So, anyway. I've finally got debugging working on android thanks to NVidia. One caveat is that this only really works on NVidia hardware; I can't debug on my Samsung phone right now. However, this is a big improvement on feeling utterly defeated, pulling out hair and swearing at unseen and presumably beleaguered tools developers.Thanks NVidia!

I'm busy, so I'd like development tools that 'Just Work'. Failing that, I'll settle for ones that 'Make sense'.

This weekend, I finally finished converting our game libraries to Linux - before now, they only ran on windows and in iOS. We've been converting to Android as well over the past month. There's something a little magical about seeing code all of a sudden working on a new platform - perhaps because it goes from 'not working' to 'oh look, a nearly finished project' much faster than normal development. Also, out of curiosity, I installed Clang and built using that, too. Amazingly, this compiled the whole project perfectly (save for a few tiny initialiser problems in a handful of files out of hundreds), gave really useful warning messages, compiled pretty quickly and and ran silky smooth. I love things that work better than you'd expect them to; it's like finding a tenner you weren't aware of. Also, I'm really excited by the possibilities of Clang and LLVM. The shared p-code thing allows for a lot of possibilities for domain-specific front end languages and compiler hacking.

Android, on the other hand, has been challenging - working with the NDK (Native Development Kit - it's what lets you write C++ on Android) is a lot less fun than developing pure Java code for android.  In fact, there should be a clue in the home page where they explicitly warn that this isn't for most people.  One set of people that it is for, although they don't mention this, is developers with large, well developed and tested engines written in C++.
  There's not a lot of 'it just works' in NDK-Land and I suspect that this is intentional. Google, Microsoft and Apple would rather we all use (respectively) Java, C# and Objective-C exclusively to each platform, ensuring an amount of lock-in. Of course, that's not what most developers want. Who wants to recreate the same work in three languages and then maintain three sets of code? Sounds like a recipe for a maintenance nightmare to me. Whilst using eclipse for Java was pretty good, smooth and even helpful (despite a few oddities and the ongoing struggle to migrate a workspace smoothly), C++ development (at least, in my android related experience and at the present moment in time) is less straightforward.
   I've found Eclipse to be very fragile for C++ development - I've had several show stopping GUI freezes on IDE startup and numerous cases where things mysteriously stopped working or gave entirely misleading and irrelevant error messages (be warned - you'll want to install plugins whilst running eclipse in administrator mode).
  Also, I'm never quite sure if Eclipse has really built all my data before sending it to the phone, although that was a problem with android development in Java too. That kind of spongy, floaty behaviour makes systematic debugging a non-deterministic chore. There's a lot that seems to go on under the hood in eclipse, lots of data squirrelled away in myriad different files. All of this makes predicting its behaviour rather difficult and I'm coding on edge, waiting for the next Heisenbug.  

 Fortunately, the command line tools are pretty good. You can export ANT command line java builds from your eclipse project - I recommend doing this even if you carry on building and debugging in Eclipse. Having a one-stop batch file that builds and deploys is no bad thing. Even if Eclipse has one its seizures, having the command line tools in place means that shouldn't stop you developing or shipping.   
   Debugging is still a challenge. As in: C++ android debugging doesn't seem to work. We had it working briefly (and slowly) but after restarting Eclipse, no dice. Following a tip from the NDK mailing list/google group, I've ordered a Tegra based tablet and will try out Nvidia's DevKit  which people have been saying good things about. For now, debugging is standard output and forced crashes.

Early, non-textured, shot of the game,
now running on Windows, Linux, iOS, Android and MacOS

I used to be more frustrated with Visual Studio and really, really wanted Eclipse to answer all the shortcomings with Microsoft's offering. However, since upgrading to a faster 64bit machine, Visual Studio has been on its best behaviour.  
  On Linux, I'm rather happy with CodeLite (which, unfortunately, usually Googles as COD elite) - it's simple, open source and tends to 'Just Work'.  Codeblocks is also fine.
  
  All in all, despite the inevitable speed-bumps that almost always come with development, I'm very pleased that we have our engine running on three new platforms and hopefully many more to come. My raspberry pi looks like it could do with a snazzy space game...
  That said, I'm looking forward to getting back to graphics and gameplay coding and also to revealing more details about our new game! 

The Eclipse IDE is full of magic. Unfortunately, I don't mean that in a good way. Whilst I do like Eclipse when it's all set up and running (right-click problem solving and their 'intellisense' actually works most of the time), setting up and copying projects is consistently a pain.
 Take, for example, special folders in an android project. When creating a new project and adding JNI support through the IDE, certain folders get magic attributes associated with them - right click the JNI folder and it knows that it should display C++ related options and android folders are also identified. However, you try importing or copying an existing project in and you'll find that your magic folders are gone - no, they're just normal folders now. I can't find where this extra information is stored but I'm still looking. Hopefully it's not in a binary format.
  I'd rather eclipse 'just worked'. If I were coding this, I'd write a more forgiving system that inferred metadata from standard names or folder contents (hey, you've got some C++ files in there, have some C++ options buddy!) rather than storing it in some external metadata which also has to be tediously maintained. That is, at least, how I do things in Clayworks.
  Context menus are great; if there's something that somebody conceivably wants to do to a file or folder then bloody well stick that in the context menu. That's what they're for. Programming is, in and of itself, a challenging activity. I've got my own problems to solve: I'd rather not waste too much of my time having to second guess somebody else's work. I would like my IDE to be easier than dicking around with makefiles, not make life harder by being opaque. I've wasted hours, days trying to balance delicate eclipse projects so that they work, for a while. I know it won't last: like a neighbour's car alarm, it'll be calm for a while but it'll be keeping you awake at night again as soon as something changes.


Edit:
There seems to be a file called .cproject that has CDT project information inside. However, copying that over to the other project seems to have no effect. Perhaps there's another file that tells eclipse that it has a .cproject file, rather than just inferring this from the fact the file exists.