Forty-Two

Keep Ops Happy - Mountain West Ruby Conf

2013-04-29T00:00:00-04:00

This year at Mountain West Ruby Conf I gave a talk entitled "Keep Ops Happy: Designing for Production", a discussion of tips, tricks, and suggestions for how to design an application in a way that makes it easy to deploy and maintain in a production environment.

The slides:

And the video:

RaidIt: Final Thoughts

2013-02-11T00:00:00-05:00

Back in May of 2012 I started an experiment to help me learn how to develop and design Rails applications to be maintainable in the long term. Specifically I wanted to figure out how to stay away from Fat-Model Massively-Coupled code bases typical in Rails applications. To ensure I gave myself an environment of learning and didn't fall into old habits through this process, I gave myself a set of rules for development. These rules were intentionally set to be extreme opposites of the typical Rails-Way development practices. The rules were as follows:

Don't start with Rails
Don't start with ActiveRecord
Don't mock anything I own
No persistence at all, anywhere
Interactors are the public API of the app

As of today my experiment is complete and now it's time to go over each of these rules, what I learned, and what I recommend going forward.

Start with Rails

I broke the first rule very early on when I realized I was building functionality before I needed it. I was building interactors according to the design I had written out in my notebook but was quickly getting stuck deciding what part to implement next. Having long preferred to start as high up as possible when developing a feature and working down the stack, starting in the middle of the stack was very uncomfortable and causing me to be less, and even counter-, productive. I had initially wanted to start without Rails so I could focus on just the application, but quickly came to understand that the User-facing view is a very important part of the application, so time for Rails to come in. Once Rails was in place and I was building View-First it wasn't long before I was deleting and rewriting the code I originally wrote, as it simply didn't work with what the app actually needed. Building from the top down guarentees you are building only and exactly what's needed.

Start with ActiveModel

You can read a more detailed account of my adventures in persistence abstraction in the previous post but if you're using Rails and you don't want to dive into full database setups yet, use ActiveModel. Rails and ActiveModel are best buds, letting you send your own non-ActiveRecord models around inside of Rails helpers and url methods with reckless abandon. I've never won a fight against Rails, and ActiveModel now exists to help people stay in Rails while not using all of Rails. All of raidit's domain models are ActiveModel entities, and Rails couldn't care less that they aren't ActiveRecord. You can even use ActiveModel outside of Rails! It's a fantastic library, and if you aren't familiar with it yet, check it out!

Mock? Don't mock? Testing is hard

Mocks and bogus objects are a great way to ensure that tests are isolating the code in question. One of the biggest pain points of most Rails applications is testing and the misuse or non-use of test isolation. In many cases, every test in the suite tests the entire stack, requiring data in the database and calling code that ends up modifying the database. This is the main cause of slow test suites, and fixing this on established applications is very difficult. Mocks can help out tremendously here but if overused they can also put the application in a worse situation: where tests pass but the application itself doesn't work.

One of the easier mistakes to make when working with mocks is mocking implementation instead of behavior. I've seen a lot of code that makes this mistake, and it is frustrating to work with. One small change to the code, without changing behavior, breaks multiple tests and updating those tests is tedious and unproductive. I'll give an example of this kind of mocking, and how to fix it. Take the following method and test:


def some_action
  @models = MyModel.where(:active => true).order(:name).limit(10)
end

def test_some_action
  where_mock = mock
  order_mock = mock
  limit_mock = mock
 
  where_mock.expects(:order).with(:name).returns(order_mock)
  order_mock.expects(:limit).with(10).returns(limit_mock)
  MyModel.expects(:where).with(:active => true).returns(where_mock)
  
  get :some_action
  
  assigns(:models).should == limit_mock
end

There is nothing here to show intent. This only tests implementation, any single change breaks the entire test even though the behavior may not change at all (for example, swapping the order and limit calls), and the test is very hard to read. This code and test is easily fixed up while still using mocks and staying away from database objects by using an Intent-Revealing Name (nothing is more important than good naming). Simply refactor that scope mess into it's own method and mock that one method.


def some_action
  @models = MyModel.active_sorted_by_name(10)
end

def test_some_action
  MyModel.expects(:active_sorted_by_name).with(10).returns([])
  get :some_action
  assigns(:models).should == []
end

Now the code shows it's intent, the tests show and prove behavior, and any changes made to active_sorted_by_name won't cause the test to fail. Testing the behavior of this new method can then be done with actual database objects and in one place only, minimizing the speed impact on the full suite.

There is one very important caveat: this pattern can only be used safely if you also have an end-to-end acceptance suite. I believe this is so important that I'm going to say it again: If you are going to use mocks in tests you must have a separate acceptance-level test suite! If there are no tests to prove that objects to actually communicate as the mocks say they do, then the mocked tests, in many cases, are actually a hinderance, passing when the application itself fails.

In raidit I made sure to have a full end-to-end Cucumber test suite to ensure the mocks in my controller tests were correct. I did still follow my original plan of not mocking what I own; considering Rails Controllers outside of "ownership" of the application. None of the domain-level unit tests use mocking, and with in-memory objects that worked really well, though please see my previous post for the caveats of this approach. I'm no longer in the "mocks are bad!" crowd but I still am careful to make sure I'm using them correctly and to have another test suite to keep my mock use in check.

Hide persistence details

Following along the previous point, and the previous post, if any code has to do with the details of how an object is persisted, hide that code inside of a method on that object, or in objects explicitly built to handle persistence logic. Controllers should never care how persistence is implemented. When these kinds of details are kept hidden from the rest of the application, the application's domain model can grow and evolve into a readable, reusable, understandable API that reveals the intent and capabilities of the application without bogging the reader down in unimportant implementation details.

Interactors, or Use OOP!

If I was to pick one definite win in this experiment it's how the Interactor pattern has helped me understand what Object-Oriented Programming and Single Responsibility Principle really means. There's nothing magical about Interactors — in almost every case they should be just plain Ruby objects — but nothing helps the human mind out more than giving concrete names to a concept. As Uncle Bob pointed out in his talk, having a set of Interactor objects leads you to organize your source code in a way that makes it almost trivial to understand what an application does. Here's Raidit's list of interactors. Even if you didn't know what the application did, you can easily find out what operations are currently available by looking at the classes in this directory, vastly improving code readability and facilitating code reuse in ways I've honestly never experienced before.

This isn't to say everything should be an interactor. As stated before I took the opposite extreme position to help prove a point. There's nothing wrong with doing simple one-liners in a Rails controller, such as #find, vs building an entire new object just to encapsulate that one piece of logic (e.g. FindRaid). As in all things, be pragmatic. If you ever get stuck wondering where functionality should go, and it doesn't fit a Controller or a Model, make a new object! That's all the Interactor pattern is: lots of small objects that each have as few responsibilities as possible.

The Problem Isn't Rails

When you get down to it, Rails does not encourage bad code. There seems to be a case of tunnel vision, where developers get stuck in the mindset that "if Rails doesn't give it to me it doesn't exist." We need to remember that Rails isn't in control, we are. Rails is just the framework, we should always strive to keep our code well managed and organized. As Kent Beck says in his book Smalltalk Best Practice Patterns

Good code invariably has small methods and small objects. Only by factoring the system into many small pieces of state and function can you hope to satisfy the "once and only once" rule. I get lots of resistance to this idea, especially from experienced developers, but no one thing I do to systems provides as much help as breaking it into more pieces.

So in all this, my suggestions for keeping your Rails application code base under control is as follows:

Keep controller actions short.
Keep models short.
Build lots of small objects, and put them in well named locations, like app/[object-type]. E.g. put decorators in app/decorators (and update config.autoload_paths appropriately).
Only put in lib/ what can be pulled into a gem. We need to stop treating lib/ as a catch-all dumping ground.
If it's a domain model, put it in app/models. The class doesn't have to be ActiveRecord::Base to live there.
And above all else: Listen to your code! Test pain is a good thing, it's telling you there's a problem with the design. Refactor until the code is pain free once more.

It's been fun to buck trends and intentionally go places I know I'd have troubles. I've learned a lot through this and discussions over code design and I hope I've helped out others in their quest to better software design as well.

What's next then? I plan on rewriting raidit in Go, writing about my experiences along the way. Go is a fascinating language that is filling the gap between low-level (C / C++) and high level (Ruby / Python) languages, with full Erlang-esque concurrency support. I highly recommend at least taking a look at Go and Mozilla's similar language Rust. I've probably spent too much time only in Ruby and need to get branching out to new languages again.

Implementing Persistence

2013-01-28T00:00:00-05:00

In Part 3 of this series of posts I talked about abstracting persistence layers and the various patterns used, as well as showing the pattern I built into raidit. I've now built the ActiveRecord-based Repository implementation, thus finishing raidit, so now it's time to talk about how this went, what I learned, and if this pattern is a good model for future projects.

The resulting implementation can be seen in app/repositories/active_record_repo.rb.

The ActiveRecord models themselves are found in app/repositories/active_record_repo/models.

Getting here unfortunately wasn't simple. This turned out to be the most complicated part of raidit, not only in building the mapping between domain models and ActiveRecord, but in dealing with bugs in the application that were hidden by the in-memory implementation. The first big issue I had to solve though was dealing with associations.

Associations

When working with in-memory objects, it's very easy to write code that expects objects and their associations to just exist. When I updated the app to serialize to a real database, these associations completely broke. Guilds no longer had their list of Raids, nor Users their list of Characters, for example. The last thing I wanted to do was build a half-baked version of ActiveRecord, so the quickest thing I could get working was to eagerly load every assocation on the requested object. This is fine for small sets of objects, but it can't possibly scale and is not easily maintainable.

Another solution I considered would require a redesign of most of the domain models to work more like ActiveRecord, where models would be able to query for their associations as they needed them, something like the following:


class Guild
  def raids
    Repository.for(Raid).find_all_by_guild(self)
  end
  
  def save
    Repository.for(Guild).save(self)
  end
end

But even with this, it's very easy to create data bugs related to memoization, or to feel you still have no real control over the queries an application is making for optimization purposes. Even so, there are further issues with this pattern.

The Empty State

As I worked on getting the acceptance tests running under ActiveRecord, I quickly realized that I had completely forgotten to implement raidit's Empty State. As in-memory data disappears on server reloads, I had added bootstrap data in an initializer to give me a good base state to play with. This hid the fact that the site broke completely for a new user with no characters and no guild. The simplicity of raidit meant this was easy enough to fix (7303934a and 188c48c5) but this could be substantially harder as an application gets more complex.

Effort vs Time Saved

It's always extra work to add an abstraction to a project. Abstractions require public APIs, converting data moving through the various layers, and implementation of the backends to handle specific details. All of these pieces require design, testing, and development. Abstractions emerge from refactoring messy or duplicate code to improve readability and maintainability, and good abstractions end up saving far more time in future development than it took to build them.

Does raidit's Repository model fit this? I'm not sure. Being able to run in-memory only means test are extremely fast (https://gist.github.com/2886208). Compared to typical Rails app tests there is a lot of time saving here, but in terms of development cost, I have yet to experience any sort of time savings. While it's nice to have a clean set of methods I can call and mostly ignore persistence details, reality proves that this kind of API constantly grows. Each request requires a slightly different set of data, which requires a new API call, a new test, and updates to the underlying ActiveRecord models.

This very quickly became Not Fun and is the reason it took me so long to finish up raidit and get this post written. A good lesson I've tried to follow throughout my development career is that if some code or feature request fills me with dread, that probably means there's a fundamental issue with the design of the code or with the current development practice(s). Functionality should always be fun to add to an application. The Repository pattern, as I've implemented in raidit, removed a lot of that fun.

So would I recommend others follow this pattern? No, not as I've done it here. There may be ways, and there are probably applications that would work well with this pattern, but I would urge caution not to build in abstractions just to have them. So what about preventing another Typical Rails App? I propose the following development suggestions.

Don't leak persistence details outside of ActiveRecord (with the caveat that column names ending up in the view isn't in itself bad). For example, any code that chains scopes and/or ARel methods should refactored behind an Intention-Revealing Name ¹ (an instance method, class method, or object).


# Change this
current_guild.raids.order_by(:created_at).limit(10)

# To this
current_guild.recent_raids

class Guild < ActiveRecord::Base
  def recent_raids(count = 10)
    self.raids.order_by(:created_at).limit(count)
  end
end

You can still have fast tests as long as you aren't putting data in the database for every test. Have a suite of tests that do talk to the database to ensure that side of the app is working, then use bogus objects or mocks elsewhere. This pattern will only work if you also have an Acceptance test suite that tests the full stack, taking the role of "Contract Tests"². Mocked tests without tests that prove that both sides of the mock are correct will leave you with a brittle test suite that hides bugs in how your objects communicate.

In explicitly taking the other extreme from ActiveRecord coupling everywhere, I've learned a lot more about what not to do and how to find a good middle ground when dealing with serialization / persistence. Use objects and/or methods to hide implementation details (like ActiveRecord scopes and ARel). This gives you names which provide meaning and intent in your code and give you the hooks to start building fast tests and improve your overall design.

I'll have one more post in this series that takes the development rules I set for myself in Rules for Rails App Development and lists out which ones are good, which ones didn't work out, and overall thoughts on this experiment.

1: Smalltalk Best Practice Patterns Get this book if you're serious about learning how to Think in Objects.

2: Integration Tests are a Scam by J.B. Rainsberger

Naming is Everything

2012-10-01T00:00:00-04:00

I would like to start this off by recommending all developers who care about building readable, well designed code go get a copy of Uncle Bob's Clean Code. This book is a treasure trove of examples on refactoring and renaming to make cleaner code.

I've come to realize that there is nothing more important to crafting good, well designed code than choosing good names. Software could be exquisitely designed, with multiple levels of tests, and a beautiful web of clean, decoupled objects and services but all that work will be for naught if another developer can't read and understand the resulting code. Software development is two processes: Informing others what the computer is doing and Telling the computer what to do. If the code fails at the first then any work done towards the second is for all intents and purposes moot. Computers couldn't care less, but the people who follow up on our work definitely do.

The most unfortunate consequence of the "get it done quick" culture is we forget our code will be read and maintained by someone else at some point in the future, and we confuse our understanding of our code with code being understandable. Why? Because the first thing get-it-quick development does is skimp on naming. We end up with classes named Controller, Manager, and Factory. We end up with God objects that sit on top of massive hierarchies and bloated Utility objects because we couldn't think of another place to put said code. We write methods with ambiguous names, or worse we leave in names that no longer match the method's functionality. We write code blocks full of single-letter variable names, use names with no rhyme or reason (enterprise = spock + kirk) or even reuse the same small set of variables requiring the reader to keep register memory in his or her head.

And yes, I have done pretty much all of this, though if I've "spock + kirk"'d up any code I don't remember, and I'm terribly sorry if I have. At least that kind of code tells the reader outright that what they are reading is nonsensical. I have also let curse words into production, something I will never, ever let happen again, but I digress...

So what am I doing to stop this madness, and what can we all do to make code readable, understandable, and maintainable?

Choose Good Names

Of course, there's one small caveat...

There are two hard things in Computer Science: cache invalidation and naming things.

-- Phil Karlton

Choosing good names is hard. Really hard, but anyone who said software development was easy was lying. Oh yes giving directions to a computer is rather simple, but as I said previously, that's a small part of what we do. As many prominent developers have been saying lately, Software is a Social Art. We are constantly learning from each other, sharing tips and tricks, yet we then turn around and drop turds of code on each other. Software is written once, maybe modified a few times, but read many times more, by people we do and do not know, and every step of our development should live and breath this fact.

Even the simplest methods are exponentially more understandable with good naming. Take a copy method, which is easier to read?


  def copy(x, y)
    x.member1 = y.member1
    x.member2 = y.member2
  end


  def copy(from, to)
    to.member1 = from.member1
    to.member2 = from.member2
  end

A simple set of names makes the method immediately and completely descriptive without even looking at the implementation. It's copying data from the object at from to the object at to. With the nondescript x and y parameters we have to look at the implementation to know which one is copying to and from where. Taking the care, the few seconds or few minutes to write out the variables in full, to name your objects descriptively, to make sure the code is readable, will pay off in spades. Spending ten minutes now will save the next developer potentially hours of maintenance work.

To help with this understanding I've pulled together a number of examples from raidit on good naming, some bad naming, and some refactoring I've done to get to better naming.

Parameters and Variables

Renaming params to attributes

Parameter names don't have to be short to be bad. Sometimes the name just has too much leeway in what it could be. In this case params is too open of a name. I don't want to be passing any random bit if data into this method through it's parameters. What I'm passing down are the attributes of the model I'm building, so lets name the parameter as such.

ChangePassword

Verbose variable names are always better than one- or two-letter names. My choice of very explicit, verbose variable names in this object leaves no room for doubt or confusion when reading this code.

Methods and Functions

Refactor #run to a static method

As I mentioned in Part 2 of this series, as well as in my post at the Collective Idea blog, I've been defaulting my Interactor objects to have a #run instance method which contains the main chunk of functionality for that action. Now I was at the time aware that run is a not a good name and as I have built out the functionality in raidit I've been finding better names for these methods. I've also come to understand and appreciate that Dude, not everything is an object and have been changing some simpler Interactors to be static methods with names that are very explicit on what they do and/or return.

Constructors

Now this may come across as an odd section given my focus on Ruby, but a constructor is nothing more than a method that builds and returns a new object. In Ruby's case, any static method that returns an instance of the class said method resides in can be considered a constructor. For example, take the following class with a few named constructors:


class ListMessages

  def self.received_by(user)
    new(:received, user)
  end

  def self.sent_to(user)
    new(:sent, user)
  end

  def initialize(sent_or_received, user)
    @messages = Message.send("#{sent_or_received}_by", user)
  end

end

Then compare the code to run them


messages = ListMessages.sent_to(user)

# vs

messages = ListMessages.new(:sent, user)

The differences are subtle but noticable. The first one reads more like a sentance, a statement of intent, while the second requires you to understand that you need to pass in a symbol stating which type of Message you want for the given user. When code tells you it's intent without getting you bogged down in the implementation details, then you know you have good naming and are progressing towards better overall design.

Behavioral Naming and Consistency

Once you're in the habit of spending time picking good names, and figuring out how to write readable code, you'll find yourself in situations where the names you've chosen are fine names in and of themselves but there's something still not right about them. You may be running into definition and naming inconsistencies. One of the less immediately obvious aspects of well designed, well named code is that the names are consistent and follow a convention across the application. If you have Create and Add that do the same thing, you should pick one and refactor the rest. In raidit, I have the following conventions for my Interactors:

Find[Type] - Return a single object of the given type
List[Types] - Return a list of objects, though that list can be empty
Update[Type] - Process an existing object with new information

For app-specific functionality I work to keep the nomenclature following the application's needs and not named according to any implementation details (such as being a website). For example, I used to have FindRaid named ShowRaid (commit) but reverted that when I realized that the action had nothing to do with "showing"; it was pulling the object from persistence. Adding a new raid is called ScheduleRaid. Creating a new account is done in SignUpUser and it's counterpart is LogUserIn (and you know, I should fix those to be either LogInUser or SignUserUp), in keeping with the wording used on the site itself: Sign Up, Log In, Log Out. With objects named like this, we now have an application structure that tells you what the application does straight from the file names. Check it out for yourself! raidit/app/interactors

As I've developed raidit, almost every time I've gotten the "this isn't quite right" feel, or whenever my test setup started to get unwieldy and painful, the most common fix has been to figure out which name is wrong. Once I've found the offending name, either via changing an existing name or extracting code from one name and giving that code its own name, the rest has simply fallen into place. Good naming may be one of the hardest jobs in software but it's also one of the most important. We must work to ensure our code is readable by other developers, not just something we understand.

Bad names facilitate bad design. Good names breed good design. Spend the time to do this part right and the rest will follow.

Abstract Persistence Logic

2012-07-13T00:00:00-04:00

Choosing a persistence library at the start of a project is almost always a mistake. As the beginning is the point of highest ignorance, why make such important and hard-to-change decisions before there's any definitive knowledge of the real needs of the application? The most important aspect of any given application is it's business rules and domain model, both of which are code. In the end, users don't care how data is persisted, but they do care that the app works as expected. Focus on the application and everything else will grow from there. Persistence is an implementation detail and thus should be a low priority task.

There are a number of benefits to deferring this decision. First and foremost is that this deferment will lead to the construction of a persistence API for your application. This API then makes any persistence requirement trivial to add to the project when it's needed. Secondly, because persistence is deferred, and because every application needs data, the first persistence implementation will be a simple set of in-memory collections. This makes the code, and more importantly the tests, extremely fast.

While I love ActiveRecord and wouldn't use anything else for SQL-based databases, Rails and the typical way Rails apps are built does not make this deferment easy. Rails wants us to use ActiveRecord and it wants us to not care where we let ActiveRecord code slip in. If I had to point at one specific habit of Rails developers that has caused the most maintenance pain, it's the hard coupling of the database details to everything in the app, especially the views and forms (quick note: use ActiveModel, it's awesome). So, to ensure we don't end up in this morass again, we need to look elsewhere for our solution.

It's not hard to find many different patterns developed over decades for solving this problem. Most of the best patterns can be found in Martin Fowler's book Patterns of Enterprise Application Architecture (PoEAA). The two I spent the most time investigating are Repository and Data Mapper along with a number of existing Ruby libraries implementing these patterns.

The pattern I chose needed to fit two criteria: simple and pluggable, as I needed to be able to have an in-memory implementation for development and tests right next to any true persistence implementation. I chose to go with Repository in raidit over DataMapper as DataMapper, like ActiveRecord, still adds too much knowledge of persistence to the models (e.g. User#save) and I want a clear deliniation between persistence and the application. The Repository pattern does exactly this. Because Ruby is so dynamic, implementing this pattern is very simple. I'll show two implementations here.

Note: I'm implementing a very basic version of Repository. Where PoEAA talks about a Criteria system (which some have interpreted as an entire library or language, like ARel or Hibernate's HQL), I want to use nothing more complicated than method calls. I don't want any code outside of a Repository implementation knowing how data is stored or queried.

The first implementation, and the implementation that raidit currently uses, consists of a single, simple object to map models to the appropriate repository implementation. As you can see here, this object is nothing more than a few helper methods on top of a Hash:


class Repository

  ##
  # Add mapping(s) to this repository. This will add to the existing known
  # set of mappings. To start the mapping list anew, use +.reset!+ first.
  ##
  def self.configure(options = {})
    @mappings ||= {}
    @mappings.merge!(options)
  end

  ##
  # Clear out all known mappings
  ##
  def self.reset!
    @mappings = {}
  end

  ##
  # Find the defined mapping for the given Domain Model class
  ##
  def self.for(model_class_or_name)
    @mappings[model_class_or_name] || @mappings[model_class_or_name.to_s]
  end

end

It's usage is simple. First define how the classes map to their persistence:


  Repository.configure(
    "User"        => InMemory::UserRepo.new
  )

then request the persistence object whenever persistence is needed:


  user = User.new
  Repository.for(User).save user

An alternate implementation that's arguably more Ruby-esque is to simply set constants to the implementations, using those constants directly as needed, like so:


  UserRepository = InMemory::UserRepo.new

  ...

  user = User.new
  UserRepository.save user

Both setups have a number of benefits outside of the decoupling of persistence. Having this intermediate layer allows having multiple different implementations of persistence, even to the point of communicating with multiple different databases at the same time if so required. This layer also ensures that the models themselves know nothing about the persistence implementation, so I can be sure that there is no leakage of responsibilities.

You can see the current implementation of raidit's various in-memory repositories in app/repositories/in_memory.rb. You can also follow along on a gist I'm continuously updating with full test suite timing runs here: https://gist.github.com/2886208. As you can see, raidit's tests are super fast, even the cucumber features!

Now, once I get to the point where I know what persistence library will work best for raidit, I'll have a fully defined API for what the application needs, letting me implement only and exactly what's needed and ensuring that the tests for these objects are segregated from the rest of the app, keeping the application tests fast forever.

Refusing to make decisions now can lead to huge benefits in how the application performs, it's architecture, and maintainability. So stop writing ActiveRecord spahetti code and take control of your application before your database takes control of you, again.

Rules for Rails App Development

2012-06-05T00:00:00-04:00

If you haven't yet, please read Designing a Rails App Part 1.

It's well known that starting a project can often be the most difficult phase of the project. You have ideas and probably some preliminary design work, but actually codifying those ideas requires a different thought process and a little bit of motivation. Often, to alieviate the difficulty of starting, you get the typical setup steps taken care of: generating a new Rails template, getting the database set up, choosing your favorite test framework, etc, giving you a familiar framework in which you can start your new project. These are the steps I need to not take if I'm going to understand where my pain is and how to better design and build an application.

In order to fully seperate myself from the "Rails Way" and the typical Rails development process, I've put in place a number of rules to guide my development and thinking. All of these rules can be summed up in one statement: The Rails Way is Wrong. Now don't get me wrong, I don't really believe that everything Rails does is bad, I'm simply giving myself an extreme stance in order to re-learn everything I've known about application development. In terms of development, I've realized that familiarity breeds laziness, and in order to prevent any old-way development from seeping into raidit, I've set up the following rules to ensure every step contains unfamiliarity.

What I hope to learn from this process is how much of the Rails Way is truely good to use vs how much of the Rails Way is due to cargo-culting, and how I can change my development practices in the future to be better at what I do in the long run. So with that, the following rules are:

Don't start with Rails
Don't start with ActiveRecord
Don't mock anything I own
No persistence at all, anywhere
Interactors are the public API of the app

Don't start with Rails

The first step towards understanding good application design and development is to stay away from frameworks entirely. It is an unfortunate truth that Rails development is a lot of cargo-culting these days, so in order to avoid this, I need to stay away from Rails entirely at the beginning.

Following Uncle Bob's talk, I've decided to give the Interactor / Entity design a try. The idea here is that your business logic is codified in Interactor objects. Interactors then interact (thus the name) with Entities to fulfill a single use case. With this I have a good top-down design start point, the Interactor, which will help me flush out my domain models. As I don't have Rails, I have to explicitly require whichever dependencies a given Interactor or test needs, ensuring dependency pain when it crops up. As a bonus, given that the tests work only in memory, they are very fast.

Starting without Rails forces you to really think about your initial design and what the application itself actually needs. You focus on individual use cases, what the major features are, and the typical actions a given user will perform on a regular basis. Once you've got a good basic design, you can start in with the tests and code. There is no set point at which you add your framework(s). For some, they can build the entire application without one, but more likely than not having a front-end drastically helps in choosing which features to implement next. I recently found myself in this situation and have thus added Rails to raidit.

Don't start with ActiveRecord

Tom Stuart recently presented a fantastic talk entitled "Stop Using Development Mode" in which he outlined the typical Rails development process perfectly. In general, we have a tendency to start with the database changes, then work up to the model, continuing up the stack developing a new feature. It turns out this is a horrible way to build software as you end up writing code before it is needed, which often leads to unused code. When you work top-down you are always building only and exactly what the application needs.

As stated in the previous section, my development then starts with the use cases, building an Interactor for each use case I lined out in my design notes. You can see all the current interactors in raidit's /app/interactors. Once I added Rails my start moved to the very top: an acceptance test to ensure that all the code I wrote was exactly what I needed.

Do note that my domain models are not ActiveRecord, and they never will be. They are plain Ruby objects that implement the non-business rules (basically code that can and should be shared across Interactors). I will add persistence to these domain models much later via another layer in the app. In this way, the tests for the domain models are super fast, and tests that actually hit a persistence layer will be in their own suite. You can find raidit's domain models in /app/models.

Don't mock anything I own

Mocking is not only a difficult thing to do right but it is also a discussion that as yet has no right answer. Some live by mocking everything you own, some by only mocking code you don't own, some hate having mocks in tests at all, and some pick and choose according to the situation at hand. There are pros and cons for all of these. I have personally chosen to never mock anything that I (e.g. the application) own and I'll explain why.

I have experienced the pain of mocking absolutely everything, which I henseforth will call "overmocking." While I understand the basic premise behind this kind of mocking — you are testing just the code and not the interactions of other objects — in my experience, the mocking ends up testing the implementation of the code and not the behavior. Changing the implementation of a method or an object then breaks a ton of tests even though the behavior is exactly the same. Coupled with bad or non-existent design, overmocking makes tests incredibly brittle, and even worse I ran into situations where the tests pass but the code was completely broken, leading me to no longer trust the tests at all.

In my quest to figure out what actually works, I'm going to take the exact opposite stance for raidit: don't mock anything that I own. This has a number of benefits: the code itself is exercised instead of a mock, the tests will tell me if something isn't implemented, and test setup will tell me very quickly if I have too many dependencies for a given method or object. One potential issue to this stance is that it doesn't exactly adhere to the "fail quick and only in one place" idea of testing, in that if a method on a domain model breaks, the tests for the interactors that use that method will all fail. I don't forsee this being a big problem but if it doesn't become more painful than I'm expecting I shall make the appropriate changes.

When I say "mock the code I own", I am talking about the application and not Rails. When I added Rails to raidit, I initially treated the controller code as part of the application, not using any mocks. This led to test pain when the controller tests needed to know explicit details of the Interactors and the domain models themselves, which felt like way too much knowledge for a delivery mechanism. Thus I am now using mocks for controller tests. The acceptance tests run the full stack, ensuring my mocks are not hurting me through false positives.

No persistence at all, anywhere

Obviously the app will eventually need persistance, but one of the quickest way to slow down your test suite is to talk to an external service, be it a database or a web service. You want tests to be fast and honestly your app doesn't and shouldn't care at all where the information comes from or where it goes. Databases are nothing but an implementation detail. The Java community has known this for over a decade, why have we regressed so far with Ruby and Rails?

So don't add persistence until you absolutely need it. Who knows, you may not need a database at all! As Uncle Bob found out with FitNess, they ended up deferring the database right off the feature list entirely. Build exactly what you need (in raidit's case, a set of in memory repositories) and no more. Your test suite and your code will thank you for abstracting out this logic.

Interactors are the public API of the app

As I mentioned in Part 1, one of the problems with Rails development is that Rails code is often written to talk to any and all objects whenever they are needed, whether it be a library, a model, a controller, or even a view. While there's nothing wrong with code autoloading — it can be a very handy tool — it has been heavily abused and used as an excuse to not care about architecture and design. I've put this last rule in place to ensure I don't fall this same trap. My unit tests will help me stay aware of dependencies as they do not load Rails and thus have to explicitly require each file needed.

Interactors are the public API of raidit. Rails code will never directly use any object that isn't an Interactor, though it may use objects that are the results of Interactor interaction. The Rails code is not allowed to access any of the Repositories directly, nor should it do anything with the domain models.

The one caveat here is that I'll let Rails code use domain model instances iff they come from Interactors. The original Interactor / Entity design called for Request and Response structures to communicate with Interactors. As we are in Ruby, I don't feel this extra step is necessary. For now, Interactors can and do return domain model instances (such as FindUser) if so required.

Through experimentation and discussions, I've settled on the following design rules for an Interactor:

Implements a single use case, no more no less
Have names that start with verbs
Receive current state information in the constructor, like current_user (see SignUpToRaid)
Can have any number of methods, and default to having #run if nothing else fits
Instance methods should be given input data as regular params (try to stay away from hash arguments)

I expect a lot of this to change over time as I write more Interactors.

So Lets get Started!

These rules will help guide and teach me real and proper application development and design. I don't know how long this will take, but I will work hard at keeping the posts coming as I learn more.

Designing a Rails App Part 1

2012-05-29T00:00:00-04:00

There is a rather frustrating problem with Rails development: nigh every Rails application quickly trends towards unmaintainability. These Rails apps become monolithic, bloated, difficult-to-understand and even harder to maintain over time. This isn't an attack on Rails itself, it is just a set of libraries, but this is a pervasive problem in the Rails community today. I myself have ended up with mess of a Rails app more times than I care to admit, and most of them were ones I started! Why do I and others constantly fall into this same trap and how can we stop this downward spiral?

When I worked at Bloomfire, I had the good fortune of getting to work with Mike Moore who pointed me to a number of videos and articles on software architecture and application design. Mike has been watching and fighting this problem for years now, but it's only recently that more developers, like myself, are starting to understand the problem and realize just how big of a mess it is we keep making. What I also didn't know until now is that thes problems were solved 20 years ago.

The one talk that really opened my eyes was Uncle Bob Martin's Architecture: The Lost Years at Ruby Midwest 2011. If you haven't watched this talk, and you write software of any kind, please watch it now. Everything that follows uses and builds from what's mentioned in this talk.

Through my research and through a ton of personal introspection I've come to realize the fundemental issue with so many Rails apps: Failing to Manage Dependencies. We as a greater development community has lost a lot of core architecture and design lessons learned in the earlier years of software development. Just because we can access any model from any other place in our application, doesn't mean we should. In the typical Rails app though, we do. We let controllers access every model in the system. We let models talk to any other model in the system. And we write tests using tools that make it so trivially easy to not care about dependencies. Who cares of one single factory actually creates 30 different models underneath? It's Ruby, it just works! Until it doesn't, and for me the point of realizing that it no longer works is far past the point of easy refactoring. Now we're in pain-ville, again.

Are your test slow, and by slow I mean does it take longer than a few seconds to run your entire unit test suite? Can you get instant confirmation that your current test passes or fails? Do your unit tests hit your persistence layer at all? Do the tests require the persistence layer?

This is what the Agile and TDD folks have been calling Test Pain for years. Yet we keep ignoring it, treating tests as something we'll deal with later. What's the major truth of tests? If you don't test now, you never will. This is just as true about keeping your test suite maintainable. More often than not, if your test suite is intensely painful, you abandon the suite instead of fixing it because fixing would require redesigning the entire application... which is what we should have been doing in the first place.

How did we miss this for so long, and why do we continue to build our applications the same way? Do we even know we are making mistakes? I have to admit that before Mike, I didn't. I only knew that something wasn't quite right with whatever app I was working on, and I would do better about it next time. Yeah, that has yet to happen.

So it's time to fix this, or at least give myself a chance to apply lessons learned from previous devs, to actually change how I do Rails development in general, and see if I can find a general path that lets me keep my code under control in the long term. To do this, I've taken raidit, a calendar app built to facilitate my WoW guild's raid scheduling, and have started it over from scratch. I'll be using this project to test, prod, and experiment with different design techniques, learning how to feal test pain, and figuring out just how in the world you go about designing software.

I have no idea how long this will take, but I shall be documenting my progress as much as possible (slight apology, I've been doing this for a month now and have just got the first post made, I'll be better about this in the future).

Please feel free to comment here, comment on the raidit repo, or get a hold of me personally. This kind of learning is not something one does in isolation. I hope that my experiences here help many fellow developers learn and vastly improve their skills. We're all in this together, lets do something about it!

My next post will focus on how I got started, Use-Case driven development, and the rules I've put in place to control and guide my development and design.

Time for a Personal Refactoring

2012-04-10T00:00:00-04:00

[a] disciplined technique for restructuring an existing body of code, altering its internal structure without changing its external behavior Code Refactoring - Wikipedia

As a developer, both career and hobbyist, refactoring is in my blood. I'm always looking for ways to improve existing code to improve its quality, testability, and maintainability. Refactoring is probably the most important technique any developer learns and one that should be constantly practiced on a daily basis. Without constant refactoring, code bases grow into massive, unmaintainable behemoths that often grace the front page of TheDailyWTF. That said, refactoring is hard in many ways, but none so hard in that it requires you to accept that with whatever you're building, you are probably doing something wrong.

Software as a field and an industry moves blindingly fast. Techniques, tools, paradigms, and even languages change on a monthly and even weekly basis requiring constant learning to keep up and to stay competitive in the market. Ideas that were Best Practices last month may be Anti-Patterns the next. New ways of thinking about problems show up daily. This doesn't mean your old knowledge is suddenly invalid, you simply need to keep adding more tools to your toolbox. Put another way, you have to constantly be increasing what you know and refactoring how you learn to stay ahead of the wave, or risk getting overwhelmed.

Now, I'm still relatively new to the industry -- I've been a professional developer for about six years now -- but I've come to realize through personal experiences and from others that the technical side of software development is only one part of this job, and a minor one at that. You may be a technical genius who can code circles around most other developers, but if you can't work with other people, if you can't effectively communicate, you aren't going to go very far. @brixen said it best recently in his talk Is Node.js Better that Programming is a behavioral science. I highly recommending reading the entire talk, but the gist is this: all software gets used by people, both who are technically inclined and those who aren't. Every line of code you write embodies your personality, and unless you never release any code you write, anyone who uses your code is interacting with you. It's vitally important to never forget this; who you are and how well you work as a team is at times more important than your technical chops, but how do you continue to improve here as well?

We need to refactor ourselves. We need to constantly be evaluating how we react to situations, how we communicate with team members and how we work with people we don't particularly know (e.g. clients of your product or users of your Open Source project(s)).

I'm not going to sugar coat this, as I'm writing this post more for myself than anything: this is very hard. As hard as it can be to keep up with what's going on technically in this industry, it's exponentially harder to recognize your own faults, much less to fix said faults. I have a tendency of being extremely hard headed and I know personally just how hard it is for me to recognize when I'm wrong, much less to have the strength to change my mind or how I act accordingly. As a developer I fall into the group of people who think extremely logically, and left unchecked this almost always causes issues.

Due to the nature of people and communication, it's impossible to take emotion out of any sort of communication, and this includes even the code you write. While you can't completely prevent anyone from taking offense to anything you do (this is the Internet, lets be real here), there are plenty of steps that can be taken to minimize potential conflict points, the first being how you communicate with your team-mates and users. What makes this particularly more difficult with technical minded people such as myself is we tend to think logically, without emotion, and with no hidden meanings. However it's unfortunately human nature to look for and find hidden meanings any and all communications. I personally need to be much more mindful of this truth in everything I do. What I do know is that the extra mental effort to ensure a given message isn't accidentally accusatory or blunt is much less than the effort required to fix problems such careless communications cause later.

So far what I've found that does help me is friends: friends who are willing to call me out when I'm being an ass and friends who understand who I am and who want to see me become a better person. It has taken a long time but I've realized now that I cannot fix or improve myself without help. I know I'm going to screw up, but as long as I have people around me who will tell me when I screw up, I'm sure I will make progress. In fact, I don't believe it is possible for anyone to become a better person without the help of others. Humans are social creatures, we can only thrive when we work and interact with others. With that, I'm taking the next step: improving my presence in the greater developer and Open Source communities.

As you've probably noticed by now, I've got a new website and a new domain. After working with Wordpress for a few years, which did the job just fine, to really rebuild and help people understand who I really am I've rebuilt my site from scratch. As a developer I like having complete control over what I'm doing and the tools I use. I've also learned to value simplicity and to abhore complexity, so a huge PHP site and a WYSIWIG text editor on a web page just isn't cutting it anymore. I've changed over to running a statically generated site using Jekyll, with the source of the site being available on Github. The design comes from the Jekyll Bootstrap theme built with Twitter Bootstrap. I can now work on the website using my two favorite tools: Vim (though right now I'm playing with Sublime 2 + Vintage Mode) and a terminal.

One huge benefit of having a static, simple website is that all of my work can now be pushed to Github. Why is this good? Because Github is my Resumé. I'm a huge proponent of Open Source and love how easy Github has made it not only in helping and contributing to existing projects but in starting and running your own, as well as making your work visible to anyone who's interested. The first place I go when I'm researching a new candidate or interviewee is their Gitub (Bitbucket, et.al.) pages to see if there's anything there and what they've done. In this industry, nothing speaks like code, because even if the person is the best communicator in the world, if they can't code at all, they aren't qualified for the job. But a great developer and a great communicator? That's what I want to be.

Along with the new website design, I've changed the domain of my website to jasonroelofs.com. As I get to know more people in the industry and more people get to either know me or start using my projects, I've ended up confusing people with my current pseudonym "jameskilton". It's not obviously a pseudonym, and it doesn't help that there are people out there who do have the name "James Kilton". I've also read that it's more professional to use your real name, though I'm not sure how much it matters these days. Either way I will be working towards retiring my old pseudonym, starting with a new website domain. I'll be updating Twitter and my email addresses soon. I'm not sure what to do about Github quite yet, as it's much harder to announce there that the account name is changing. Suggestions are welcome.

So hi! I'm Jason Roelofs. I love Ruby and I love Open Source. I'm extremely hard headed and can come across as an ass at times. I ask that you not take such incidents personally, but to call me out on it. Thanks!

Manage Resque with Upstart and Monit

2012-03-12T00:00:00-04:00

Resque, the unix-y process and child based background worker library from Github, is a joy to use. It's simple to set up and to integrate into your code, only requiring Redis for the queue, and runs through a single rake task from the command line. It really couldn't be easier for how scalable and robust the library is. What isn't easy, though, is managing and monitoring your Resque workers in production. There have been numerous attempts to solve this problem, including god.rb, Monit, bluepill, resque-pool and foreman, but from my experience all of these solve only 90% of the problem, and that last 10% is guaranteed to cause you pain either through fragility, complicated configuration, or not alerting properly when your workers disappear. I've come to believe that any tool that tries to handle both process management and monitoring and alerting is trying to do too much and can't satisfactorily do either.

At Bloomfire we've been using god.rb to manage and monitor our Resque workers for a while now, but it's never been great. We quickly realized that we couldn't expect the god process itself to always stay alive, so we hooked up Monit to monitor god (you can imagine how many jokes this spawned...). This worked well enough until we had to upgrade our REE install to 2011.12 to close the String#hash security hole. With that upgrade, god.rb started very randomly segfaulting somewhere deep inside of Thread. Most of the time this crash would leave the Resque workers running and god.rb would pick up where it left off, but sometimes it took everything down in the crash. From the reports I've heard so far (via Issue #81) REE 2012.02 doesn't fix the segfault problem, so it's time to move away from Ruby-based solutions entirely. Enter Upstart.

Upstart

Upstart has actually been around for a number of years. If you're running any recent Ubuntu build even as early as 10.04 LTS, you are already running Upstart. In short, Upstart manages your system, mainly through keeping services running but also through managing important system configurations. If you've ever dreaded writing yet another SysV init script and making sure your service handles it's own daemonizing, Upstart is the answer to all of your prayers. This service is the culmination of decades of system administration experience. It knows what you need to do, lets you do it very simply, and it doesn't get in your way. If you're curious just how much easier it is compared to SysV, check out the difference between Upstart's SSH and the SysV SSH script (from Ubuntu 10.04): https://gist.github.com/1987878. Not only is the Upstart script vastly simpler, but Upstart will also make sure the process stays running! If ssh dies for whatever reason, Upstart will know and will restart the service almost immediately. Try getting SysV to do that! On top of this, Upstart is completely reliable. It already manages the majority of core services on Ubuntu machines, so chances are, you're already relying on 100% uptime from Upstart.

Before we dive into the actual Upstart config file, do note that this is for Ubuntu 10.04 LTS and thus an old version of Upstart itself. The newer versions of Upstart include better handling for running processes as different users, so when 12.04 LTS comes out, I'll be updating the scripts to use these new features.

/etc/init/resque.conf

The following is all you need to configure Upstart to run your Resque workers:

For more detailed explanations of what each individual command means, please consult the Upstart Cookbook.

There's a lot going on here so we'll take it a few lines at a time. The first three stanzas description, start, and stop should be rather self explanatory.

The two restart lines are what tell Upstart that it needs to try to keep the process running. In this case if 5 restarts happen in 20 minutes then something is very wrong and needs fixing. I chose this broad window due to the time it takes for a Resque worker to load the Rails environment before running. Tweak to your config as necessary.

One of the beauties of Upstart is that it understands running multiple instances of single service through the instance clause. With this clause, instead of having multiple resque.conf files filling up /etc/init, running multiple instances of a service merely requires a parameter, in this case ID, which Upstart will use as a unique identifier to manage the running process.

The script stanza is where we define how the process itself is run. Everything in this block is bash and is run in its own process. In this script, we simply exec our usual rake environment resque:work with the required environment variables as with any other management solution. As the version of Upstart that comes with Ubuntu 10.04 doesn't yet support running processes as users other than root we have to explicitly set permissions and run the Resque worker as another user. If you're already running Upstart 1.4 or later, you can use setuid and setguid. When we upgrade to Ubuntu 12.04 I'll update this post with the changes.

The duplicate pidfile handling in this script block is required for our monitoring to properly watch for each running Resque worker. This script is also working around a timing issue in the current version of Resque (Pull Request #529). So, to keep our monitoring tool from going haywire while Resque workers restart, we first write out the pid of the Upstart script process (through $$). We then tell Resque to write out its pid out to PIDFILE once it's loaded, replacing the pid of the now defunct Upstart process and allowing monitoring to pick up the actual Resque process for memory and cpu load monitoring.

You run your Resque worker with one simple command:

start resque ID=0

If you need to have more Resque workers running, increment ID and run the command again. As long as ID is different for each run, Upstart will treat each call as a new process and manage it accordingly. Once you have your Resque workers running, it's time to configure monitoring.

Monit

The only thing Upstart doesn't do that's vitally important to any production environment is downtime alerting. While the respawn limit stanza ensures that Upstart doesn't get into an infinite loop trying to restart a broken service, Upstart will not warn you when it decides to stop restarting. Monit adds this layer of security to your upstart configs. I know I mentioned Monit in the first paragraph of this post in the list of tools that don't quite do what's needed, and I should clarify what I meant about Monit. Basically, we were never able to get Monit to manage and monitor the Resque workers, mainly because at the time Resque did not write out it's own pidfile, and thus Monit had no way of finding the process. Resque has since added the PIDFILE parameter which should make pure Monit solutions easier if you prefer to go that way. That said, Monit is, in my opinion, the best single-server monitoring tool around, which is why we've chosen it for monitoring the Resque workers, and why I recommend it no matter how you're running your Resque workers.

Our current Monit config is the following (it's ERB and is pulled directly from our Chef configuration):

You'll notice that we're technically managing our processes in Upstart through Monit. It is possible to put Monit in "monitoring only" mode here, removing start program and stop program and having Monit only alert when things go down. Doing so will require you to figure out another way of restarting your workers as needed, such as a Rake task that sends SIGQUIT to all current workers known to Resque. We've chosen to use Monit to restart our workers due to the ease in which it is done, a single command -- monit -g resque restart -- restarts all currently defined workers. If this setup gets untenable I'll update this post as such.

There is one minor issue with this setup related to the PID swapping described above. Every time you restart the Resque group you will get emails for each Resque job's "Action Done" event as well as a PID swapping alert "PID Changed" when Resque finishes loading and finally writes out its pid. When the Resque pull request gets applied, the PID swapping errors should go away, but I don't think it's possible to stop the "Action Done" emails without halting other potentially important emails for other services you might be monitoring. I'm up for suggestions or ideas.

And that's it! You have Resque running under Upstart and monitored by Monit. While that was a lot of discussion over a couple of configuration lines, there's a lot of underlying knowledge required to make the most use of these too tools. Hopefully I've covered enough for you to be comfortable working with this setup.

Moving a Rails app from MySQL 5 to Postgres 9

2011-10-16T00:00:00-04:00

At Bloomfire we recently decided to do a full migration over from MySQL 5 to Postgresql 9. There are a number of reasons for this, including scalability, better feature sets (such as Psql's full text search engine) and getting away from Oracle before we are forced to move. For the sake of this post, detailed reasons aren't important.

What I've put together here are the few issues we ran into and changes we had to make to get the application working under Postgres. Overall the code changes are minimal as we mostly use ActiveRecord code for communication, and in the places we write SQL directly it was mostly standards compliant. That said here's what we did have to change.

TEXT and :limit

With MySQL, TEXT fields have a maximum length, and as such MySQL offers multiple types of TEXT fields. ((http://dev.mysql.com/doc/refman/5.0/en/storage-requirements.html#id816304)) PostgreSQL's TEXT field will auto-scale the field to fit whatever data it needs to contain. ((http://www.postgresql.org/docs/9.1/interactive/datatype.html))

Fix: Remove any :limits on TEXT fields.

Type Conversions

MySQL is very loose when it comes to dealing with types. If a field is an integer, you can send in a string and MySQL will convert what's needed, and vice-versa. PostgreSQL however will complain if you pass in the wrong type.

Fix: Make sure you're using the write types in your queries.

SUM(boolean_filed)

When you ask MySQL to SUM() a boolean field in a query, it happily returns the number of rows that are TRUE. This is because MySQL stores TRUE and FALSE as 1 and 0 respectively, thus SUM is simply a mathematical accumulation. PostgreSQL has a dedicated BOOLEAN column type, so this no longer works.

Fix: Change your SUM to be explicit about TRUE and FALSE:

sum(CASE boolean_field WHEN TRUE THEN 1 ELSE 0 END)

AS operator

MySQL has an implicit "AS" operator in it's query parser, e.g.

select alias.some_column from table_name alias

PostgreSQL wants you to use the AS operator.

Fix: select alias.some_column from table_name AS alias

TIMESTAMP fields

Postgres has much finer precision with TIME fields. Where MySQL works with seconds, Postgres saves down to the microsecond ((http://www.postgresql.org/docs/9.1/interactive/datatype-datetime.html)).

Primary Key Sequences and resets (for Tests mainly)

Postgres in some cases doesn't auto-reset your primary key sequences. When this happens you'll run into tests that die with a unique key constraint error. Still working on the best way to solve this but for now you can do the following for each table in which you need the sequence reset:

ActiveRecord::Base.reset_pk_sequence!(table_name)

SQL Function Equivalents

MySQL: WEEK(), MONTH(), YEAR() PostgreSQL: EXTRACT({day, month, week, year} FROM TIMESTAMP '...') ((http://www.postgresql.org/docs/9.1/static/functions-datetime.html))

MySQL: LIMIT offset, row_count PostgreSQL: LIMIT row_count OFFSET offset

GROUP BY

It appears that MySQL does extra work underneath when dealing with GROUP BY. I ran into an issue where Postgres was complaining about missing fields in GROUP BY clauses. After doing some research, it seems that Postgres is strictly following the SQL spec where MySQL is not, so where MySQL auto-fills in the required GROUP BY columns if they are missing and can be inferred, PostgreSQL requires that you specify every GROUP BY column required in the query.

ORDER

Where MySQL defaults an ORDER BY to be the ids of the rows found, Postgres has undefined default sort order. ((http://www.postgresql.org/docs/9.1/interactive/queries-order.html)) If your tests or code expect a certain order and the query doesn't have an ORDER BY clause, your code will be nondeterministic. Adjust accordingly.

Data Migration

The easiest way to move data over from MySQL to PostgreSQL is to use ActiveRecord. While MySQL does have a "--postgres" option in mysqldump, it doesn't really do the job it needs to, requiring some text manipulation of the data before PostgreSQL will accept it:

Other than that, and changing to the postgres gem, the conversion to PostgreSQL has been relatively painless so far. I'll update this post if more issues come up.

Component / Entity Systems

2011-03-17T00:00:00-04:00

Note: I've put this work on hold and am going another direction with this project. The C++ code is found in the ogre branch. I'll have a new post about what I'm putting together soon!

If you've done any sort of game development in the past 5 years you've most likely heard of "Component Systems" or "Entity Systems" for doing object and scene management over an OOP hierarchy. Object Hierarchies may start out clean, but over the course of the development of a game, become a very difficult to work with. Components -- very succinctly built self-contained clusters of data and logic that can be added to, or taken away from, anything -- are the answer.

There is a ton written about these kinds of systems. A quick Google search of either of the terms in the title will get you to many of the most popular posts describing the why of Component systems, so I'm not going to dive into that myself. Instead, this post will be explaining how I'm implementing Components in Project Slartibartfast, why I've made the decisions I've made, and how I plan on continuing development with this system.

As an avid supporter of Open Source software, it's been annoying reading so much about this radically different methodology and find very little code, or code examples that don't solve the issue I'm trying to solve, concerning Components. Now that I've sat down and actually turned an idea in a working system, I realize that these systems really do end up being very specific in their implementation to the product at hand, be it a game, an engine, or something not game related at all. So this isn't per-say a "this is how you implement Components and Entities" but an exposé of how I've decided to implement this in my game.

As an aside, because these arguments come up so often, my personal view of software development and design is that pragmatism beats idealism every single time. As long as you have a good reason for your decisions, build something that works, don't fret that it's not a "best practice."

That said, lets dive into the Component system of Project Slartibartfast.

When I sat down to start designing the base system this game will use, I needed to answer two questions: Where does the data live? and Where does the logic live? These questions will have vastly different answers depending on who you talk to, what they're trying to build, and what language they're working in. Do the components know about the data? Does the Entity have data and Components the logic? Are there other systems that handle the logic as well as the connecting of Components to Entities? It's important that you can put an answer to these questions before you start coding because once you've started building a system following your answers, changing your mind later usually results in a very large rewrite and big headaches getting everything running again.

I decided to take as pragmatic an approach as possible and build a system that's simple but no simpler, easy to extend, and doesn't require a bunch of crazy C++ syntax or templates. I came up with the following.

Actors

Actor is what I'm naming my Entity construct. My first real foray into modern game development was UnrealScript in Unreal Engine 2, which uses the Actor terminology and it's just stuck with me. Actors are very simple, they only hold the list of Components that define them. Every Actor has a TransformComponent by default which defines how they hook into the world (an idea I pulled from Unity3D). Due to this fact, the TransformComponent is defined directly on the Actor itself and not in the components array.

Actor.h

Components

Components are pure data. There is no per-frame logic in these. Each component knows what data it needs to keep track of for the system to work. This data can include publicly usable data (position, rotation, scale) or hooks into various subsystems (like a SceneNode or Camera in Ogre). Because these classes are pure data, they're in most part just a collection of public attributes.

Implemented Components

When a Component is added to an Actor via addComponent(), the Actor ensures that the Component is registered with it's appropriate Manager. The logic for Component <-> Manager registration is handled with the REGISTER_WITH macro that's added to each Component. After fighting for a while to make Managers and Components all OO happy with templates and polymorphism I blew the entire system away (don't you just love 300 page error reports due to gcc template errors?) and implemented a few macros to define some helper methods common across all Components and Managers. This turned out to be much, much cleaner and easier to work with.

Managers

Managers are logic. If there's a Component in the system, it has a related Manager to, well, manage it. Components get initialized by their Manager when given to an Actor. The Manager has update() for any per-frame updates that need to happen for the Components it's in charge of. Managers also keep internally a list of all of their Components. Only Components added to Actors are considered "live".

Managers Definitions / Managers Implementation

You'll notice that there are also macros in place here: MANAGER_IMPLEMENTATION and MANAGER_DEFINITION. These, like the Component macros, define common functionality across Managers for dealing with Components. With this I'm able to keep a very specific list of Components for each Manager instead of fighting with polymorphism, templates, and types.

Also, Managers are Singletons. There's going to be quite a few of them as this project progresses and it was the only idea that made sense.

API

With all of the above now in place, the API for using Actors and Components couldn't be simpler. It's straight C++ object handling:

Everything here just works. The Ogre::Camera gets initialized, attached to the Actor's scene node to inherit rotations and translations. The Input is hooked up to provide a full 6 degrees of movement in two parts, MovementComponent handles the keyboard ESDF mappings, and MouseLookComponent of course handles the mouse input, and the Actor is placed into the world at the given location.

Adding a Planet to the game is just as easy:

where we get a new Actor, set its scale through the TransformComponent I talked about earlier, then adding a Mesh representation to have something visual on the screen.

Continuing from here, there is one convention that I break really badly that I wouldn't mind finding a cleaner way of handling. If you'll look at CameraManager.cpp you'll see that I utterly beat and abuse the Law of Demeter. This is one aspect of Component systems that is probably the least "solved," for lack of a better word: communication between Components. An Ogre::Camera, for this system to work, needs to be directly attached to the SceneNode representing it in the world. This SceneNode is on the Actor's TransformComponent, so I have to go component -> actor -> transform -> sceneNode to get at that information. Since this is the simplest solution that works and doesn't really get in my way I don't really have a problem with it but if I do come up with a better solution I will be sure to post about it.

And that's really it. I'm pretty happy with the system so far, it's easy to understand, handles communication automatically so I don't have to remember to connect anything, and is very easy to work with and extend. If any serious issues pop up with my decisions here, I'll be sure to update this blog as necessary.

Project Slartibartfast - Introduction

2011-03-09T00:00:00-05:00

It's been some time since I last wrote (a little over a year, yikes!) and with a new year I've decided to finally take on some projects that I've been putting off for far too long, as well as getting back into writing and keeping a log of what I'm working on and why. I've spent so many years, and so many hours researching, reading, and consuming writings and code related to game design and graphics techniques that it's time for me to give back to the community what I've learned over the years, and what I'll be learning as I progress through this next project of mine.

I've been a gamer since I got my hands on my family's first computer back in the early 90s. Running Windows 3.1, I was immediately enthralled with the device, and when I discovered things like Minesweeper and Solitaire, that was fun, but the game that hooked me was Treasure Mountain! If it wasn't for my parents constantly yelling at me to get off and give my sisters time to play I would have sat there forever. Thinking back, it is a good thing that my parents got on my case about playing games, as I've learned to pace myself, but they were never able to dull that need to play something. Then I discovered programming.

I attribute my love of programming to Texas Instruments and their graphing calculators. I had messed around with QuikBasic (Nibbles!) but it didn't click until I was teaching myself TI-BASIC on the TI-83. From there I was able to teach myself C to build apps for the TI-89 using TIGCC, and with these tools at hand, I was making my own games!

From then on I knew I was going to be a programmer, and initially wanted to get into the games industry, though for plenty of reasons that are a post in and of itself, I ended up choosing to stay out of that realm. I spent plenty of time tinkering, researching, and just keeping up with game development, but I never really sat down and made a game, or spent enough time on one to consider it "finished". This year it's time for that to change.

And thus was born Project Slartibartfast.

I've always been a huge fan of Science Fiction and Outer Space (favorite books of all time: the Dune series by Frank Herber, followed by the Foundation series by Isaac Asimov) and as such I've always looked for and played games in that setting. Outside of the classics, like Descent and the Wing Commander series (which sadly I never did get around to playing), and modern games like the X-series and EVE Online, there really isn't a whole lot out there actually set in outer space. The game I played the longest was definitely Freelancer, but even that was a flawed gem of a game; so many good ideas, but so much it could have been and wanted to be. There's also Infinity: The Quest for Earth but that's a long way from being released, and it might be too realistic to actually be a good game in the end, though that's something we'll have to wait and see.

As I cancelled my World of Warcraft account last month for what is probably the last time, I started thinking about the next game I wanted to play. It needed to be something that wasn't fantasy (in the MMO realm, it's either fantasy or post-apocalyptic, neither of which are terribly appealing to me right now). There's Black Prophesy but that's more like an MMO Counter Strike in space. There is no exploration, everything is instanced, and it feels very small. There's also Jumpgate: Evolution, the successor of the cult hit Jumpgate, but that game is so mired in politics and a black-hole of information that I've basically given up on it until it comes alive again. EVE Online is simply too much for me, I'm looking for a game I can play on and off as I want and not forced to dedicate hours of time (woo WoW raiding!). Then the thought hit me, "screw this, I'm going to make the game I want to play." And thus here I am.

Project Slartibartfast will be a game, set in space, in a procedurally generated galaxy. You explore systems, gather materials, set up trade routes, build transit networks, and eventually do this all with other people. It's hugely ambitious, I know, but hey, reach for the starts, right? I'm basically taking as best I can all the features I loved about the various games I've played or researched to make the "perfect" space-based game.

What's with the name? Well if you haven't been able to tell by now, I'm a huge fan of Douglas Adams and the Hitchhiker's Guide to the Galaxy and as such a lot of what I do is someway or somehow a reference back to this series. The name comes from a part where Arthur Dent ends up stranded on the legendary planet Magrathea he runs into a very strange man

  "You chose a cold night to visit our dead planet," he said.
  "Who ... who are you?" stammered Arthur.
  The man looked away. Again a look of sadness seemed to cross his face.
  "My name is not important," he said.
...
  [Arthur] looked at the old man, his face illuminated by the dull glow of tiny lights on the instrument panel.
  "Excuse me," he said to him, "what is your name, by the way?"
  "My name?" said the old man, and the same distant sadness came into his face again. He paused. "My name," he said, "is Slartibartfast."
  Arthur practically choked.
  "I beg your pardon?" he spluttered.
  "Slartibartfast," repeated the old man quietly.
  "Slartibartfast?"
  The old man looked at him gravely.
  "I said it wasn't important," he said.
  The aircar sailed through the night.

As I don't really have a name, story, setting, or in fact anything really set yet with this game, other than it's in space, the name Slartibarfast felt like a perfect fit. Who knows, maybe that's the name that will stick for good, wouldn't that be something?

As usual my work is up on github. I'll also be tagging all related posts on this blog with the slartibartfast category to make them easy to find.

My next post will be about the component system I've decided to implement, stay tuned!

How Rice Works

2010-02-23T00:00:00-05:00

Rice is one impressive piece of software. I highly doubt I could have written this from scratch (and for that I give Paul Brannan huge accolades for his work) but after working with and on this library for at least a year I probably could. Now that lead maintainership has been passed on to me, I'd like to take some time to explain exactly how Rice works, how one can dive in and help develop this library, and what my plans are for the near future as I work to make Rice and the rb++ suite; a full and proper replacement for SWIG when wrapping C++ libraries into Ruby (if you're wrapping a pure C library, I highly recommend Ruby-FFI, though Rice will work).

So with that, I'm putting together this post to help clarify exactly how the various parts of Rice work together, how the entire process flows and more importantly helping anyone who's wanting to learn and work on Rice with a solid starting point (or at least officially crooked). After reading this, anyone should be able to understand the various classes in Rice, and have an idea on where certain functionality lives when looking to add features, or address any bugs. So with that, we'll start at the top.

What is Rice?

Rice is a wholly remarkable piece of software that takes complicated C++ classes, methods, types, etc and exposes them into Ruby using as simple an API as can be made. In short, you can take a C++ class such as:


class Generator {
  public:
    Generator(int seed) : mSeed(seed) { }
    ~Generator() { }

    int getRandomInt() { return 4; } // chosen by dice roll, guaranteed to be random

    void setSeed(int seed) { mSeed = seed; }

  private:
    int mSeed;
};

and use in in Ruby seemlessly:


g = Generator.new(5)
g.random_int          # => 4
g.seed = 10

through a very simple Rice wrapper Ruby extension:


void Init_generator() {
   define_class<Generator>("Generator").
      define_constructor(Constructor<Generator, int>()).
      define_method("random_int", &Generator::getRandomInt).
      define_method("seed=", &Generator::setSeed);
}

Were this class to be wrapped using Ruby's C-API, every method on Generator, and even the constructor would have to be wrapped in a C method with calls into Ruby's API (particularly Data_Make_Struct and Data_Get_Struct), very tedious and time consuming work, and at times quite error prone. So how does Rice make this tedious process as simple as four lines of code? C++ templates, and lots of them. We'll go through each line here and dive into Rice itself to see what's happening, but first, a quick word on Rice's file naming convention.

ipp, hpp, cpp, _defn ... what??

Because Rice is so template driven, most of the code cannot be pre compiled. That which can gets put into cpp files and ends up in librice.a, otherwise the core of Rice is template code generation which means only the code needed for the given extension is generated and compiled when building the extension itself. Given this, Rice still tries to strongly adhere to the separation of definition and implementation code. The definitions of classes are normally placed in [class_name]defn.hpp, with the implementation placed in [class_name].ipp (to keep it separate from compilable code which gets placed in .cpp files). These two files are then combined in [class_name].hpp which is meant to be used in Rice extensions when including various pieces of functionality. When working inside of Rice and you need class definitions, you'll want to use #include "[class_name]defn.hpp" in most cases or risk running into circular dependencies and "already defined" compiler errors. Otherwise, when writing an extension using Rice, you'll #include "[class_name].hpp" to ensure that all functionality is available for the extension.

As for the .mustache files, a lot of Rice's template code is very repetitive and so instead of using complicated macros like Boost.Python does, Rice simply uses Ruby to generate the C++ code. Rice currently uses the Mustache library for this and the code for this process can be seen in rice/code_gen.

Defining a class: define_class

To expose a C++ class into Ruby you first need to define the class, give it a name, and tell Rice which C++ type this class is (as a side note, you can create a Ruby class not hooked to a C++ type, just don't give define_class a type). The prototypes of define_class are found in Data_Type_defn.hpp. The one we are particularly interested in is:


//! Define a new data class in the default namespace.
/*! The class will have a base class of Object.
 *  \param T the C++ type of the wrapped class.
 *  \return the new class.
 */
template<typename T>
Rice::Data_Type<T> define_class(
    char const * name);

Following the code into Data_Type.ipp, we can see the implementation of this method:


template<typename T>
inline Rice::Data_Type<T> Rice::
define_class(
    char const * name)
{
  Class c(define_class(name, rb_cObject));
  c.undef_creation_funcs();
  return Data_Type<T>::template bind<void>(c);
}

This method starts out creating a new Rice::Class with the given name, making it a subclass of Object (aka a top-level Class, emulating what Ruby itself does). A quick jump into Class.cpp shows us that define_class(char*, VALUE) simply forwards off to the Ruby API call rb_define_class, then takes the resulting VALUE and wraps it into a Rice::Class.

Rice then undefines #alloc and #initialize to prepare the class for Rice's own type management (to be discussed later). The third line of this method is what actually binds this new class to the C++ type, binding it as a top-level class (Data_Type::bind is templated to the superclass type, so void in this case).

Data_Type binding

Probably the most confusing, and at the same time most important, aspect of Rice is how it manages C++ types and their link into the Ruby world. Every C++ class that's to be wrapped into Ruby must be bound to a Ruby class which is done in the static method Data_Type::bind (implemented in Data_Type.ipp). This logic forms the core of how Rice does automatic conversions from C++ types to Ruby types and vis-versa.

To start, every Data_Type<T> has a static pointer to a Ruby class (Data_Type<T>::klass_). After checking that this C++ class isn't already bound to another Ruby class, bind() registers the Ruby class with the Ruby GC and sets up a Caster between this C++ type and the Ruby class (to be discussed later).

We've now got our C++ class Generator properly bound to the Ruby class Generator and can continue.

Adding methods to a class: define_method

We're now at the point where we can start wrapping the methods on our Generator class. We'll follow the code execution of wrapping the method Generator::getRandomInt(), starting with Module::define_method found in Module_impl.hpp.

Quick side note: Rice's C++ API attempts to mimic Ruby's top level Object structure, so the interplay between Data_Type, Module and Class can get confusing. Suffice it to say, class-based objects in Rice are all Rice::Module-s at the top, which then itself is a Rice::Object.

The specific define_method we're interested in here is as follows:


  //! Define an instance method.
  /*! The method's implementation can be any function or member
   *  function.  A wrapper will be generated which will use from_ruby<>
   *  to convert the arguments from ruby types to C++ types before
   *  calling the function.  The return value will be converted back to
   *  ruby by using to_ruby().
   *  \param name the name of the method
   *  \param func the implementation of the function, either a function
   *  pointer or a member function pointer.
   *  \param arguments the list of arguments of this function, used for
   *  defining default parameters (optional)
   *  \return *this
   */
  template<typename Func_T>
  Derived_T & define_method(
      Identifier name,
      Func_T func,
      Arguments* arguments = 0);

The Rice::Arguments object has to do with the implementation of default argument definitions and I'll get to that later.

This method is the beginning of where the true template craziness (genius? probably) happens. As a warning, if you are not comfortable with templates, the following is either going to confuse you, or it will open your eyes to a whole new level of possibilities in C++.

Rice works directly with Function Pointers and template definitions tailored to react to specific function pointer definitions to generate the appropriate code for properly wrapping C++ methods into Ruby. As a refresher, a C++ function pointer looks like this:


ReturnType (ClassName::*methodName)(ArgT1, ArgT2, ...);

so if we were to print out the function pointer of Generator::getRandomInt we would see:


int (Generator::*getRandomInt)();

This is what Func_T ends up being, a function pointer type. It can be a method with any return type, and any number and type of arguments, all handled by this one method. Looking at the implementation of this in Module_impl.ipp, we can see that execution is simply forwarded off to detail::define_method_and_auto_wrap given the Module/Class/Data_Type (*this), the name, the function pointer, any defined exception handler (this->handler) and the arguments object.

From this point, Rice needs to make a very important distinction on the method being wrapped: is it a class method, or a static method? The method Rice::detail::define_method_and_auto_wrap, defined in detail/define_method_and_auto_wrap.{hpp,ipp} uses Wrapped_Function classes, via wrap_function (detail/wrap_function.{hpp,ipp}) to make this distinction and to set up the final method object before finally jumping into Ruby's C-API for finally exposing this method.

Auto_{,Member}_Function_Wrapper

Opening up detail/wrap_function.ipp you'll see a very long file full of very repetitive method definitions. The key here is this method uses templates to match the passed in function pointer to one of Auto_Function_Wrapper (static functions) or Auto_Member_Function_Wrapper (a class instance method). Due to how Auto{,Member}Function_Wrapper works, wrap_function also has to have template specifications for functions with a return type and functions with void, as well as specifications to handle const and non-const function pointers.

To continue from here we need to understand how Rice actually hooks into Ruby. Glancing in detail/Auto_Function_Wrapper.hpp we'll find template-overload classes similar to what's in wrap_function. These classes are the core that handle argument and return type management as well as taking execution from Ruby and passing it into the related C++ function or method. What we're going to focus on here is the ::call static method, though we'll get to the details of this method's implementations later.

Ruby's API will only take C-style functions with the signature VALUE(*)(...). The Auto...Wrapper classes are the glue that gives us a function with this signature:


static VALUE Auto_Function_Wrapper::call(int argc, VALUE *argv, VALUE self);

If you're familiar with Ruby's C-API, then this signature should look familiar, it's the -1 arity signature used when you want to support a variable number of arguments (in Rice's case, default arguments). So how does a static function on a templated class work with an instance of this class to call the correct C++ method or function? It's time to dive into detail/method_data.cpp, which defines define_method_with_data() as seen in the second part of define_method_and_auto_wrap() (Rice::protect is a wrapper around rb_protect to catch any exceptions and handle them gracefully).

detail/method_data.cpp

I'll start this section with the comment block describing define_method_with_data():


// Define a method and attach data to it.
// The method looks to ruby like a normal aliased CFUNC, with a modified
// origin class.
//
// How this works:
//
// To store method data and have it registered with the GC, we need a
// "slot" to put it in.  This "slot" must be recognized and marked by
// the garbage collector.  There happens to be such a place we can put
// data, and it has to do with aliased methods.  When Ruby creates an
// alias for a method, it stores a reference to the original class in
// the method entry.  The form of the method entry differs from ruby
// version to ruby version, but the concept is the same across all of
// them.
//
// In Rice, we make use of this by defining a method on a dummy class,
// then attaching that method to our real class.  The method is a real
// method in our class, but its origin class is our dummy class.
//
// When Ruby makes a method call, it stores the origin class in the
// current stack frame.  When Ruby calls into Rice, we grab the origin
// class from the stack frame, then pull the data out of the origin
// class.  The data item is then used to determine how to convert
// arguments and return type, how to handle exceptions, etc.

Bear with me here as I still have a hard time keeping this straight in my mind. To help understand what exactly is going on here, I've made a graph showing all the relevant links between Rice, C++ and Ruby as made by this method.

The steps define_method_with_data takes:

Create a new class inside of the class we're exposing (called the origin class).
Build a new two-element array that has [ 0xDA7A, VALUE pointer to Auto...Wrapper class]
Give the origin class an easily recognized name (which is also an invalid Ruby constant name, preventing it from every being accidentally used in Ruby code, but it prints out in object dumps on occasion).
Set this origin class to be FT_SINGLETON which hides it from the user ((http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/336728)).
Set a hidden ivar on the class that's the array built in step 2
Define a method on the origin class that links to Auto...Wrapper::call with the requested name

At this point we've got everything ready except for actually being able to call the method from Ruby. These next steps were hard to show right on the graph because Rice actually modifies the symbol table of the base class in question.

Create an alias on the origin class that aliases the method to "dummy"
Find this new alias entry in the origin class' symbol table
Modify the symbol table of the base class to call "dummy" on the origin class on any call to the method being wrapped.

We've now got everything hooked up: we have the c-style function hooked into Ruby and we have a way to access the actual Auto...Wrapper class once call() gets called. Now, you may be asking why this is such a complicated process when it could be done with a simple rb_define_method on Auto...Wrapper::call(). The answer to this question is show in the image above, there's data and information saved in this Auto...Wrapper object that Rice needs to be able to properly convert argument and return types, and handle any exceptions that may get thrown.

With that, we're done with the wrapping and exposing path through Rice. Now it's time to head back up the chain and see how Rice handles calls from Ruby and passing execution back up into C++.

Auto_..._Wrapper::call()

It's time to take a detailed look at this method. In short, once execution enters this method, it's now Rice's job to convert VALUE arguments into the appropriate C++ types, execute the wrapped method / function, then do any required conversion of return values back to VALUEs for Ruby, while also making sure that any exceptions that throw along the way are properly caught and handled (an uncaught exception in C++ will give you a SegmentationFault or a BusError. Always fun to debug).

So, first things first, we need to get back the Auto...Wrapper object we saved when we wrapped this function. This is done with the detail::method_data() function, details of which are easily seen and understood in detail/method_data.cpp. With this we run our arguments list through rb_scan_args then run through each argument and check if there's was a default set in the wrapper and if we should use it (Side note: the sanitize<> template is a no-op right now and can be ignored. It's there to attempt to clean out any references or const arguments that may come in to get the base type, but this was causing crashes on other wrappers. For now I assume there's a possibility of a small memory leak here related to reference types). Once all the types are converted it's time to actually call the C++ method or function, and handle the return type.

The noticeable difference between Auto_Member_Function_Wrapper and Auto_Function_Wrapper is that Auto_Function_Wrapper has some special handling for the "self" parameter. This functionality is here to support the ability to wrap C functions that are written in an OO way, such as:


Jukebox* newJukebox();
void play(Jukebox* jukebox, Song* song);
void stop(Jukebox* jukebox);
void addSong(Jukebox* jukebox, Song* song);
// etc.

Because of the special self handling of Auto_Function_Wrapper, these methods can be wrapped into a Jukebox class very simply:


define_class("Jukebox").
  define_method("initialize", &newJukebox).
  define_method("play", &play).
  define_method("stop", &stop).
  define_method("add_song", &addSong);

Other than this, the two classes are identical in functionality.

Constructor

Because constructors aren't accessible via normal function pointers, Rice has to implement special handling of wrapping class constructors for proper instantiation. This is done through the Rice::Constructor class and the define_constructor method. The path here is mostly equivalent to define_method. Opening up Constructor.{hpp,ipp} you can see the same ::call() static method, with a much simpler body. Every Ruby object has a free slot accessible through DATA_PTR. It's in this slot that C++ class instances are saved and used for type conversions as described next. Constructor::call creates this object and saves it in DATA_PTR for later use.

Default Arguments

The code for handling and managing Default Arguments is pretty easy to read, but how the syntax work is definitely cause for confusion. Pulled from Boost.Python's implementation of this exact feature, this functionality uses the comma operator to string together Rice::Arg objects into a single Arguments object that's then passed into the Wrapped_Function object for that method. Due to the fact the comma operator doesn't even come into play if there's a single argument to a method, I was forced to copy some functionality around to make this feature work in all situations. So where you see two methods, one which takes a Rice::Arguments and one which takes a Rice::Arg, that's what's going on. I hope to clean this up drastically in the near future.

to_ruby / from_ruby

The last piece of Rice's functionality is the type conversion system, exposed to the user through two methods: to_ruby(), which takes a C/C++ type and converts it to a VALUE, and from_ruby() which does the opposite. The signatures of these methods are simple:


template<typename T>
T from_ruby(Rice::Object x);

template<typename T>
Rice::Object to_ruby(T const & x);

For most basic and fundamental types, the to_/from_ruby implementations are very simple and can be seen in to_from_ruby.ipp, but this is only half of the story.

Rice's real strength comes in handling custom-defined C++ types and seamlessly converting them from Ruby to C++ and back again, making sure to handle inheritance (both C++ <=> C++ and C++ <=> Ruby) appropriately. The core of this functionality is found in Data_Type::from_ruby in Data_Type.ipp, and in the Rice::Caster class found in detail/Caster.hpp, but it begins at the end of Data_Type::bind, which creates a new Caster for the newly bound type:


  detail::Abstract_Caster * base_caster = Data_Type<Base_T>().caster();
  caster_.reset(new detail::Caster<T, Base_T>(base_caster, klass));
  Data_Type_Base::casters().insert(std::make_pair(klass, caster_.get()))

Caster is at the core very simple. It's a class that enables Rice to build a tree of Casters that define how to cast from a Ruby type into a custom C++ type. The code above is simply building a new Caster linking the new type to the Ruby class, specifying any superclass type as needed, then registering this Caster with Rice's Caster tree. From here it's time to look at how Rice uses this Caster list in Data_Type::from_ruby() (which btw is called through ::convert() in detail/from_ruby.{hpp,ipp}).

In the case that the C++ type we want is exactly the Ruby type we've been given, casting is easy, make a new Data_Object for that type and get the pointer. If this doesn't match, then we need to start heading up the inheritance heirarchy (rb_mod_ancestors) looking for the lowest class in the hierarchy bound in Rice, get it's Caster and attempt to cast into C++. This functionality allows any depth of inheritance to work seamlessly in Rice.

The Future of Rice

Rice is currently a very capable library, and is being used successfully by a couple of projects, but it does have a long way to go to be the SWIG replacement (along with my rb++ suite) I want it to be. For the next release of Rice I'm current planning on the following work:

Refactoring Auto{,Member}Function_Wrapper, Constructor, wrap_function, et.al. to be much simpler. Going to try to abstract out the actual method dispatch into a class like Auto_Functor_Wrapper which will handle all the to_/from_ruby and argument management magic.
Default arguments on Constructor. To do this Constructor needs to act more like an Auto_Function_Wrapper, and thus would be much easier to implement once the refactoring above is done.
Beginning into a def_helper (from Boost.Python) feature. Right now the implementation of default arguments is pretty messy. As I need to handle multiple arguments as well as a single argument, I've resorted to two define_method implementations to take these two cases. With some template magic, I can make it work with a single argument, as well as opening up a single place for call and return policy functionality, instead of multiple places that would be required today.

As always, I'm available over Github messages / Issues, Email, comments here, or on Rice's mailing list (rice AT librelist DOT com).

Modern Christianity and Evolution

2009-10-04T00:00:00-04:00

I read a very disturbing statistic recently: 39% of Americans believe in Evolution. I found this surprisingly low number in a recent article about a new movie on Charles Darwin's struggle with his life, his faith, and his Origin of Species here:

http://www.telegraph.co.uk/news/worldnews/northamerica/usa/6173399/Charles-Darwin-film-too-controversial-for-religious-America.html

The article does make some very good points. In America, religion rules. This can be seen especially today with the massive political upheaval of the recent elections, where we've got the "religious right" (mainly Christian, which I'm most familiar with) against the "secular left" fighting it out to the bitter end. What I have been trying to understand is why do less than half of all Americans believe in Evolution and how can we solve this? Is there a glaring and unrecoverable conflict between Evolution and Christianity? Is there any scientific evidence supporting either side? Is this an irrational, fearful reaction in the face of the unexpected and the new? And more importantly, why is it most Americans are so against Evolution while almost everyone else in the world has happily accepted the theory?

What is Evolution?

To make sure we're on the same page, I want to specify the definition of 'believe' here. I'm going with Webster's second definition:

b : to accept as true, genuine, or real ." ((http://www.merriam-webster.com/dictionary/believe)),

When I talk about belief in Evolution, I'm simply talking about the acceptance of the scientific evidence in favor of Evolution, nothing religious.

Also, this post will be mostly on Biological Evolution, but will touch on the general idea of Young Earth and the Big Bang Theory.

Lets start with a quick primer on what Evolution actually is. First laid out in Charles Darwin's Origin of Species in 1859, it's the theory that states (paraphrased):

Biological evolution ... is change in the properties of populations of organisms that transcend the lifetime of a single individual. The ontogeny of an individual is not considered evolution; individual organisms do not evolve. The changes in populations that are considered evolutionary are those that are inheritable via the genetic material from one generation to the next. ((http://www.talkorigins.org/faqs/evolution-definition.html))

The Theory of Evolution is the theory that species change over time. This change manifests itself in random mutations of the DNA defining an organism, which is passed down to offspring. If these mutations are harmful, the offspring don't survive. If these mutations are neutral, or even beneficial to the organism, it survives to pass down these changes to it's offspring, a process known as Survival of the Fittest. Following this process over thousands and even millions of years results in the diversity of life we see today.

Evolution today is used to explain a myriad of observations we've made in the biological world, from why there are hundreds of different species of pigeons, some slightly different, others completely different, yet definitely all pigeon, to why virii are impossible to vaccinate against and why cancer is such a brutal and difficult disease to fight.

So if Evolution is so successful in describing the biological we live in, why isn't it more accepted here in the States, and in some cases attacked as a tool of Evil? Lets look at the Christian counter-argument known as Creationism.

Creationism

I used to be a Creationist. I was brought up in a rather conservative Christian home, and as the prevailing view of the Beginning of the Universe was Creationism, I naturally accepted and followed these beliefs. What follows is what I used to believe, and what I know about the movement.

Creationism has very simple roots: Genesis 1:1 (NIV).

In the beginning, God created the Heavens and the Earth.

What follows in verses two to the end of the chapter is a day-by-day account of what God created on each day, creating the entire Earth and everything on it, in six days. Creationists take this account literally, in that it took six days, as we define "day," for God to create the Earth and all living things. They also believe that, as each day starts with "and God said...," we all appeared here one day by the whim and will of God. Picture it as a blank sheet of paper, and start drawing animals, plants, and people. There was nothing, then there was everything.

Creationism's biggest beef with Evolution is that Evolution seems to take God completely out of the picture. They commonly argue that Evolution says we're "all here by accident" or "we're a product of random chance." They claim that because of this, Evolution goes directly against the Bible and thus (in the most extreme of cases) claim that Satan introduced Evolution into the scientists' minds to drive people away from Christ. Past this, the next most popular argument Creationists have against Evolution is the idea of a biological part being unevolvable, such as the eyeball, saying Evolution cannot be true because the eyeball is so complex that it has to either exist, or not exist, there is no half-way-point because, according to Evolution, the part would have been deemed bad and been phased out many generations ago.

Where do I stand in this?

I'm still a Christian, but I can no longer accept the Creationist claims. I owe this change-of-mind to two books: Language of God by Francis S. Collins and Only a Theory by Kenneth R. Miller. These books showed me that one can believe in Evolution, and still be a Christian. For anyone on the fence, these are great books to start with, and what follows is a quick run down of the major arguments for Evolution, and how they fit into the Christian world-view.

Problems with Literal Bible Interpretations

Christianity has a very poor track record when it comes to literal translations of the Bible. The most poignant, and relevant, example would be the stories of Copernicus and Galileo. They had the audacity to suggest that the Earth actually revolves around the Sun, contradicting a long standing teaching of the Church that the Earth is obviously the center of the Universe, because the Bible says so. We all know how that one turned out.

So following this, are Creationists making another mistake in taking the Bible literally? It would appear so, as the first two chapters of Genesis give two (arguably three) different creation accounts.

Genesis 1:1 -

In the beginning, God created the Heavens and the Earth.

Gensis 1:2-31 -

Day 1 - Light and Dark
Day 2 - Land and Air
Day 3 - Sea and Land, Vegitation
Day 4 - Sun Moon and Stars
Day 5 - Water Animals and Birds
Day 6 - Land Animals and Man
Day 7 - Rest

Genesis 2 -

Earth and Water
Man
Vegitation
Animals
Woman

While the comparison isn't one-to-one, it's easy to see that the order of events do not match up. If one is to attempt to take the Bible completely literally, this is a difficult contradiction to overcome.

Other arguments against literal interpretations of the Bible include discussions of the Hebrew writing style, mainly that they wrote more in poems and stories, and that the writers were less concerned about factual accounts, but as I am not knowledgeable in this I'll leave it at that.

Evolution's "Random Chance"

A major misconception that is proving very difficult to dispel is this idea that Evolution is a completely random process. This idea couldn't be farther from the truth. Most Evolution nay-sayers completely forget about, or do not completely understand, the concept of Survival of the Fittest. All mutations are put through the most strenuous of tests: surviving and reproducing. When mutations pass this test, they get fully integrated into that gene pool and thrive. When the mutations fail the test, the organism(s) die off and the mutation is lost. The key point here is that most mutations fail the test due to the random nature of the process mixed with the very complex nature of DNA.

Due to this, were our evolutionary process to be done over, we most assuredly wouldn't be exactly the same, but fish would still be shaped like they are (for efficient movement through water) and birds would have wings and probably hollow bones (for light-weight and moving through the air). The limits imposed on organisms by the world they live in are what drive evolution failures and successes, we are not the direct result of a few thousand random mutations, we are the result of a select subset of mutations that survived among the millions that allowed us to survive and thrive in our environment.

Unevolvability

Another common argument against Evolution is this idea of "unevolvability." Creationists say that there are biological components (the eye and bacteria flagellum, to name two) that simply cannot have evolved because if a single part is missing, the component is useless. Useless parts, of course, are not favored in Evolution and thus, these parts should not exist.

This is actually a very true statement, which lends towards it's strength in the Creationism world. However, there is a very pivotal assumption being made in this argument that Evolutionists have found to be a false one. This is the assumption that the parts of these components are only useful in the component they're found in today. I'll paraphrase the argument made in Language of God on how a bacteria's flagellum actually did evolve to show this point.

Flagellum ((http://en.wikipedia.org/wiki/Flagellum)) is a collection of proteins used to propel bacteria through it's environment in a rotating and/or whip-like manner. This is obviously a very complicated system, and if any of the many parts were to be missing, this locomotion would not be possible. This does not disprove Evolution though, because scientists have actually found parts of the flagellum being put to completely different uses ((http://en.wikipedia.org/wiki/Evolution_of_flagella)). For example, scientists have found a similar sub-construct being used as a secretion system.

I'll admit this can be hard to understand for those of us who aren't medically versed, but the point is this: the process of Evolution periodically takes a working and capable part, and mutates it to be something completely different. There is a lot we do not know about this process, but there is very strong evidence that biological constructs previously thought to be unevolvable are in fact a direct product of the evolution of various other constructs. For further information, here's Wikipedia's page on the evolution of the eye.

What Evolution Does Not Cover

Given all this, there are still a few points that Evolution and Science in general does not yet have answers for. Among these are:

How did life itself get started from a lifeless ball of rock orbiting a burning ball of gas?
The Big Bang. Something exploded, but where did that something come from?

If we can figure out how to answer question #1, that would be the biggest breakthrough in all of Science: the ability to create life. This of course has some very far reaching moral, religions, and philosophical issues to work through, for if Man can create life from lifelessness, does that new life have rights like the rest of us? And what does that mean for Man itself, the ownership of a life form, etc?

As for question #2, I'm not sure if that question can be answered scientifically. As our viewing technology gets more sophisticated, we can see closer and closer to Time Zero, but how do you see anything before the Big Bang and is there anything to really see?

What I Believe

So given all this, how exactly does my Christianity work with current scientific knowledge? I believe that God used and continues to use Evolution in our Universe (even outside of the Earth? we may never know, but there's nothing stopping God from such a thing). I believe that there had to be something that God created out of the nothing that was the Void, and that this thing eventually exploded in the Big Bang.

I believe that at a certain point in our Evolutionary history, God had on his hands beings that were advanced enough intellectually to understand Him and to worship Him. He embodied these beings with souls and named them Adam and Eve, starting what we know call the Human Race.

I believe that God gave us Science and infinitely inquisitive minds and as such it's impossible for Science to in any way disprove the existence of God. We are simply learning about the world that He put us in. Any scientific finding that seems to go against my personal beliefs should be a cause of reflection on the discovery and on the beliefs I hold, not of immediate rejection of the science in question.

Going Forward

So what can I do to help fix the state of affairs in this country? I can educate people. Talk with friends, discuss with family, and encourage people to branch out. I've found that the best way for me to strengthen my faith is to doubt what I believe. With doubt comes uncertainty, and the mind hates uncertainty, and will crave resolution, which requires more information. When I can take this one further and can get other people to doubt their fundamental ideals, the discussions and debates that follow can only help to strengthen my beliefs. If I can get a few people to change their thinking, they'll do the same with others.

I realize that fanatics and extremist will always exist, but the key to defeating them is to ignore them, to defuse their movements. If they're yelling at people, and no-ones listening, they'll eventually go away. This won't be easy, but it's necessary if this country is going to stay the leader in scientific advancements.

Evolution is the process God put in place to bring about life on this Earth.

Game design lessons learned from World of Warcraft - Part 3: PvP and PvE

2009-06-22T00:00:00-04:00

So I'll be honest. I've had a huge rant building up for some time now on how Blizzard has royally screwed up WoW, and continues to do so, because they have a flawed look at how to balance PvP and PvE gameplay together.

I'm not going to write it because frankly, I'd never want to read it. So I've condensed my point into three simple words:

BALANCE

PVP

FIRST

You simply cannot have a PvE game and add PvP on top of it. PvP is player skill, PvE is just numbers. One could go to the extreme and say that they can't co-exist, but I'm not quite convinced this is true. Close, but not quite.

Adobe Flash 9 & 10 Security Requirements

2009-06-03T00:00:00-04:00

For those of us messing and working in Flash, the transition from Flash 8 through to Flash 10 have been rife with some major changes in the security model that allows a local Flash application to communicate with the outside world. Adobe, knowing that these are big changes, had its developers write pages, pages, and pages of documentation describing the where what how and why of these updates.

Problem is, it's very hard to find the simple How-Tos for common situations in all this text. I spent many days scouring and tinkering to figure out the security solution to my application: a flash-based video viewer that would connect into a server and display Motion-JPEG frames that get streamed in.

Hopefully I'll help save someone else the same hours of research and playing with this problem. My solution is such:

You first need up a simple service on the server you want to connect into that serves the crossdomain.xml content that defines your security policies.

I use a modified version of the service found here: http://www.lightsphere.com/dev/articles/flash_socket_policy.html. It's updated to be a full Daemon process, so you need Proc::Daemon and Proc::PID::File installed.

Please note:

<site-control permitted-cross-domain-policies="all"/>

is the required addition to make this script work with Flash 10. Also, if you are worried about corporate firewalls blocking ports < 1000, simply change $port to something bigger, and add the following to your Flash client:

And you're good to go!

The GOP is precisely that...

2009-05-27T00:00:00-04:00

... Grandiose and Old.

I have to say, I'm impressed. I was appalled at the atrocities perpetrated by the Bush administration, especially now as more information gets leaked out, but what's going on now is mind-boggling at best, and at worst will decimate our country to where we'll never recover.

I refer, of course, to the constant Republican QQ-ing and stonewalling about the choices the Obama administration is making.

Ever time I hear a prominent Republican talk on TV, on the radio, or read something on the Internet, I hear nothing but "waaah waaah I'm not getting my way anymore so I'm going to make life difficult for everyone who doesn't see things my way, waaah waaah!" Ironic, this, as it's the old-folks that are acting like 8 year olds who aren't getting their way.

Aparently, some of these people need a refresher for what THEY had a hand in for the past EIGHT YEARS:

DOUBLING our National Debt from ~$5 trillion to $10 trillion
TWO wars, one justified one not

The justified war in Afghanistan has been under-manned and assumed to be a "quick romp" to clear out the Taliban. Guess who's back in power and even stronger?
The unjustified war in Iraq has destroyed the friendships we had with many countries around the world and has made even MORE people angry at America, leading to MORE terrorists. Iraq will fall into civil war between Sunni and Shiite as soon as we leave, just you watch.

Decimation of personal liberties through acts like the Patriot Act and the Digital Millennium Copyright Act (What did Benjamin Franklin say about safety and liberty?)
Secret prison camps. Torture guidelines, Guantanamo Bay, Detention without Trial
... and who knows what else we'll find out about in the coming years

You guys have done a TERRIBLE job at running the country. Why should we trust, or even listen to, anything you guys have to say? You complain about lack of bi-partisanship, but then you don't ever budge on your stances on bills. You guys are going to fight the new Supreme Court nominee (well, you're probably still pissed that the exiting justice, David Souter, made you guys look like idiots, but I digress), only because it's Obama's choice, not one from your party.

You guys might mostly disagree with Limbaugh, but come on, you are all working towards Limbaugh's ultimate hope, the "failure" of Barack Obama.

Do you care about this country? Or do you only care about your personal "values" that get thrown out of the window anyway once you're in power?

Congratulations on losing most of the young vote (like myself), that's really going to help you get back into power in the future. Oh and grats on alienating the Hispanic vote, and most of the Black vote too. In fact, the women vote is going to the Democrats as well (thank you Limbaugh!)! If anything, the self-inflicted implosion of the GOP has been very entertaining to watch, but the delusion that the party is still as relevant now as it was 20 years ago is causing far more damage than good.

That said, there is a simple solution to this problem:

Grow Up.

Game design lessons learned from World of Warcraft - Part 2

2009-05-01T00:00:00-04:00

While Part 1 touched on abstract decisions affecting game design (and admittedly turned into a rant by the end, I'm kind of bad at that), part 2 here will focus on a few of Blizzard's design decisions for WoW that we can learn from, and in Part 3 I'm going to touch on my ideas and observations on the game's PvP and PvE.

Now of course I can't list, or even think about, most of the design decisions that have helped towards making WoW the massive success that it is, but there are a few decisions, both good and bad, that have hit close to home for me, decisions that I have to work with or around every time I play. Were I to be on the WoW dev team, these are some of the things I'd be recommending.

Stat Equality

Wow is all about gear because character power advancement is built completely around the stats. The stats in WoW are Strength (STR), Agility (AGI), Intellect (INT), Spirit (SPI), and Stamina (STA), and they are directly related to how powerful your character is, or at least they should be.

In WoW, stat allocations over the various classes has always felt arbitrary, and frankly it's a very complicated system that doesn't need to be, which has lead to serious balancing issues between Melee and Caster classes that have yet to and probably will never be fixed.

For example, a Warrior gets two Attack Power (AP) (stat on which all abilities scale, thus directly related to damage output) per point of strength. Rogues and Hunters get their Attack Power from AGI (Hunter is considered a "Ranged Melee" class by the game). However, Mages get a 1% increase chance to crit every 60 INT along with 100 mana per one point of INT, while Spell Power comes purely from items themselves. SPI gives a DPS boost to Warlocks because of a skill that class has, but anyone else just gets mana regen (as of 3.1, SPI does give very small amounts of crit now too), making it a very important stat for some healing classes, but not all.

Confused yet?

When WoW first came out, progression for DPS caster classes, especially Mages, stopped hard at level 60. The game did not have the stat Spell Power, so the only benefits Mages got from gear was higher health in STA and more mana / crit with INT. The game did have Attack Power, so melee were outscaling Mages exponentially, who were simply able to cast spells longer. Since then, caster DPS scaling has been an issue.

A much, much simpler system of making INT behave like STR would have gone a long way towards simplifying scaling from the beginning. If your game is going to have stats, make them consistent. Arbitrary decisions will lead to big problems, and sooner than you'd expect.

All stats need to scale to appropriate levels and then some

Following the above, make sure those stat decisions actually do scale into the foreseeable future. It is surprising to see a Blizzard lead designer explicitly say that they check the scaling of all classes only through to the next patch or expansion (which ever is nearer) and fix problems as they crop up. I can't begin to imagine the amount of work that would have been saved had more care been taking at the beginning to make stats less complicated and properly scale for longer than just the next tier of gear.

Character resource decisions can make or break balancing

The Death Knight uses a Runes and Runic Power resource system. The DK expends Runes on certain attacks, those attacks build Runic Power, from which the DK then uses other abilities that deplete the Runic Power. Any rotation a DK choses can be kept going forever.

Rogue energy is always refilling and simply serves to put a "cast time" on rogue abilities. Rogues can fight forever.

Warriors have Rage. Warriors gain Rage when they hit with white attacks or get hit. They then use this Rage to use special attacks. This system, while dependent on Rage income, can be kept going forever.

Casters use Mana, and can run out. They cannot fight forever. This is a very poor decision that has caused numerous problems with fight balancing, character balancing, and makes PvPing with these classes especially difficult.

Healers are not included in this. Healers running out of mana is a very valid 'soft enrage' system to put on a raid. If a healer could go forever, then any fight is easily trivialized.

Warhammer got it right: every class uses a refilling energy system like the WoW Rogue.

Incentives drive player inclusion

Converse to that, disinsentives do drive players to not participate. When you're dealing with a group of people, it really is quite easy to predict what that group will do. For example, by adding Battlegrounds and gear rewards for doing those battlegrounds, Blizzard completely destroyed World PvP. Why spend hours ganking people in the world when at least you get rewards for doing it in a Battleground?

Achievements!

Nothing has refreshed WoW like Achievements. Giving people the ability to show off their exploits, to be rewarded for taking on hard content, or for committing acts of crazy mindlessness (see: The Loremaster) helps keep players busy with tasks they otherwise have not done, along with just straight keeping them in-game and paying. Nothing helps a game's population than more ways for people to show off what they've done.

World Events

Special time-specific events are always a big hit for players, but they usually are very time consuming to implement and it's rare that every player gets to participate in said events. WoW's system of having set events that happen throughout the year, every year, help to make sure that as many players as possible can experience this specially crafted content. Hooking achievements to world events was of course a brilliant idea, and has made players actually look forward to the next event, preparing and planning.

Mistakes have been made, but it cannot be denied that Blizzard hit on the recipe for hugely successful MMO. It's easy to get into, easy to play, difficult to master, and full of content for everyone, from the PvPer to the PvEer to the players that just hang out with friends. There's a lot to be learned from this game, and I'm sure designers will be using WoW as a template for many years to come.

Part 3 will be my (hopefully not to angry) rant on the decision Blizzard made to try to turn a PvE game into a PvP game, and the constant balancing work that has and is going into both sides of this game.

Embedding IRB into your Ruby application

2009-04-02T00:00:00-04:00

A feature I realized would be very helpful to have in rb++ is a debugging console. Basically, I wanted the ability to drop into an irb session and have available to me the internals of rb++ and rbgccxml, all set up for the sources I've specified for the given extension.

Turns out, this is a lot harder to figure out than it would initially seem. The irb code is quite complex and nothing there is documented in any way shape or form. Also, searches around the Internet came back with results that were some six years old, and while helpful, did not pertain to my particular case.

Then I remembered that ruby-debug does exactly what I'm looking for. A quick glance in that code got me the code snippet I needed.


require 'irb'

module IRB # :nodoc:
  def self.start_session(binding)
    unless @__initialized
      args = ARGV
      ARGV.replace(ARGV.dup)
      IRB.setup(nil)
      ARGV.replace(args)
      @__initialized = true
    end

    workspace = WorkSpace.new(binding)

    irb = Irb.new(workspace)

    @CONF[:IRB_RC].call(irb.context) if @CONF[:IRB_RC]
    @CONF[:MAIN_CONTEXT] = irb.context

    catch(:IRB_EXIT) do
      irb.eval_input
    end
  end
end

And it's use is quite simple. When you want to drop into an irb session, you simply do


IRB.start_session(binding)

and you will have a code prompt with all data available at your finger-tips.

See the related rb++ commit

You use this feature in rb++ as follows:


ruby your_extension_file.rb console

From there, start playing around with the parsed source tree using either @parser or self.namespace to get started.

Game design lessons learned from World of Warcraft - Part 1

2009-03-28T00:00:00-04:00

Given that I love gaming, and I love programming, it's only natural that I put the two together. Now while I haven't yet gotten deeply into game development and design, I have become acutely aware of design decisions made in the games that I play, and I like to take mental notes of what works, and what doesn't. I also spend a lot of time watching game-related news and lurk the GameDev forums to keep up-to-date on the latest game development trends and advances.

I figured, now that I've got a blog, it's time to start taking more permanent notes on the design decisions, both good and bad, of the various games I play. To start, I'm going to pick on the 800lb Gorilla in the room, World of Warcraft (known here on out as WoW). This is the first of what will be a multi-part series of articles, as I tend to delve in to rants and ramblings at time. However, I will try to keep it all coherent. So, without further ado...

I've been playing this game on and off for nigh three years now. I've seen plenty of ups and plenty of downs, been a part of quite a few guild breakups, and have developed an almost unhealthy love of criticizing and critiquing every little change Blizzard make to the game. I'll admit, it's not the best state of mind to be in, and many-a-time it has eaten away at me and caused me to loose all enjoyment of the game as I end up only able to focus on what's wrong and not what's fun.

What follows are the results of my brooding over the what and the why of the decisions made at Blizzard. Why does Blizzard make the decisions they do, what are the ramifications, and what can other game designers and developers learn from one of the best companies in the business. To start off, the most important rule about game design in general is quite simple:

You are building your game. Make the decisions you think are right.

I give Blizzard HUGE applause for what they have accomplished with WoW. It is by far the most complicated game ever created, and the fact that they have kept it running, continually added more, and still kept the same general WoW atmosphere and Blizzard Polish since release is a testament to the courage the development team has in shaping this massive game over time. Given all the flak they receive on a daily basis on their official forums and forums across the world, Blizzard rolls right along with the punches.

If you want your game to be successful, trust your instincts. It's your game, not theirs. Make the game you want to play. That said...

Some players will know the game better than you

An interesting phenomenon I've been following with WoW is the statements of theorycrafters (those players that constantly calculate and run all the numbers they can get a hold of, to find the best, the worst, and everything in between) contrasted to the official statements of Blizzard, and what I've noticed is this: the theorycrafter community is almost always right.

As an example (and this will be difficult to really show without forum posts, but those are gone so please bear with me), with the most recent expansion to the game, The Wrath of the Lich King, we'll look at how WoW's PvP game became, quite frankly, a failure.

Wrath of the Lich King came with three changes that worked together to completely dismantle the game's PvP.

Up the level cap to 80
Restart everyone on even footing as of level 71
Introduction of the Death Knight

Upping the level cap is a normal move for any MMO expansion, and follows with the previous 10 level jump we saw in The Burning Crusade (TBC), the first expansion. There needs to be significant increases in the status of player characters, so it stands to reason that a level 80 character would easily tromp a level 70 character.

Restarting on equal footing is almost a necessity given WoW's PvE end-game. Now, first and foremost, WoW is a gear driven game. Everything you do to make your character more powerful involves gear. To help be better in PvP, you get better gear. To do better, and progress farther in PvE, you acquire better and better gear. The side effect of this is, a level capped character with full end-game PvE gear is far more powerful than a character who just hit the level cap. With a new expansion, and 10 more levels to grind through, Blizzard could not require high end PvE gear to continue through the new content or they would have very quickly lost a large portion of the player base (as only about 10% of the population really experience much of the end-game content).

So, the new area, Northrend, had to offer quest reward items and random world drop items that were almost equal in power to the previous end-game PvE gear, putting all characters back on an even footing. This means that everyone suddenly does more damage, has more health, and is overall significantly stronger (example: my best raiding DPS in TBC was around 1500. When I started raiding again in Lich King, I was already doing 2500 and quickly improving). In general, the calculations showed that a level 80 character started out about 2.5 to 3 times more powerful than at level 70.

The introduction of the Death Knight was long in coming, and heavily anticipated. A new class for WoW was just what the game needed to bring a fresh breath of air to the game some people had been playing for 3 straight years by then. It is a class with a novel new resource system (Runes and Runic Power), tons of new spells and abilities to master, and an entire new starting area built just for the class and its lore leading up to Lich King. Blizzard put everything into this class, they wanted it to be a huge success, and quite a success it has been, and a major sore spot as well.

With the triage of these three major changes, WoW's PvP was doomed from the start. It didn't take the theorycrafters long to crunch the numbers, to play with classes on the Test Realm, and to come to the conclusion that at level 80, characters were either going to do way too much damage, or they weren't going to have high enough health pools to soak up the huge damage increase. Posts on the official forums and post on respected theorycrafting forums like Elitist Jerks showed, argued, and asked constantly for Blizzard to re-think their stance on character power at level 80, but Blizzard would have none of it. If there's one statement seen the most, it was "We will just wait and see what happens" often followed by "We think it will be ok, characters will have enough health."

Well the "wait and see" has happened, and the game's PvP devolved to the worst it's ever been. Healers gave up trying to heal, because you can't heal someone who gets killed in 3 seconds or less. Entire classes stopped playing because they either couldn't do the damage others could, or they couldn't get out of the way of the incoming damage. It was, and in most cases still is, absolute mayhem. Skill has been shoved into a trailer and gear and choice of class is the king of PvP now. To see what I mean, here's the top 10 ranked PvP Arena 3v3 teams as of the March 24th:

The top 10 teams of the 2009 Arena Tournament’s online qualifier as of March 24, 2009 are listed below:

Rank. Team Name	                Classes
1. well then                        Death Knight, Paladin, Warlock
2. BARKSDALE CREW             Death Knight, Paladin, Warlock
3. monkey attack squad        Hunter, Mage, Shaman
4. GET MONEY GET PAYCE     Death Knight, Hunter, Paladin
5. Walrus Attack Squad	        Hunter, Mage, Priest
6. Paradorn’s Team	        Hunter, Paladin, Warlock
7. NO LIGHTNING GEN BRAH	Death Knight, Paladin, Warlock
7. Nekos’ Team	                Death Knight, Death Knight, Paladin
9. faceroll tunnel vision         Paladin, Priest, Shaman
10. Wd’s Team	                Death Knight, Paladin, Warlock

Pretty obvious to see just how inbalanced the PvP game has become. The top teams, while good at PvPing anyway, are so much better simply because of the class composition of the team. Entire classes are absent of this list: Druid, Warrior, and Rogue, which ironically enough were always present in Arena teams during TBC.

Anyway, it's a lot of talk to make a point, but the severity of the failure because Blizzard didn't listen to the theorycrafters at all needs to be shown. There have been many steps since taken to "tone down burst damage" since Lich King's release, but Blizzard has a long way to go. More time up front, and a willingness to admit that they were wrong, and Blizzard could have made huge strides towards preventing this catastrophe. Blizzard has been working hard since the release of TBC to make WoW's Arena a viable "e-sport", but with current state of Lich King's PvP, they have taking a very severe blow to that work, making it that much harder to convince people that Arena is worth standing on the pedestal with games like Counter Strike and Quake.

Continue with Part 2

Blog Named!

2009-03-26T00:00:00-04:00

That's right, I decided on a name. Being a huge fan of Douglas Adams, and especially the Hitchhiker's Guide to the Galaxy, it seemed only right to pay homage to such a great author and fantastically weird story.

So the choice of theme made, the name came down to two: Forty-Two, otherwise known as The Answer, or Slartibartfast, because the name really doesn't matter. I was leaning towards Forty-Two, and with an extra vote from a good friend of mine, it became the name.

I'll probably update other parts of the site to fit the theme, like the subtitle and the categories. All in all, feels good to have a solidly named site.

Now, what do you get when you multiply six by nine?

There are two hard problems in Computer Science...

2009-03-21T00:00:00-04:00

Cache invalidation, naming things, and off-by-one errors. I have no idea what to name this blog, so for now, it's going to remain un-named until either a good suggestion comes along, or inspiration hits me.