bookmark_borderA good use for ":group" in Rails

Recently I needed to make a custom view for activity logs in my Ruby on Rails app. I wanted to create a summary of the activity grouped by the day, without any duplicates. e.g. if I performed an action an object twice, I didn’t want it to be listed twice.

So how does one do this in Rails? Simple – use the :group (i.e. GROUP BY) parameter when doing a find() to combine results into single results.

The nitty gritty

For reference I used the following schema on my logs:

create_table "application_logs" do |t|
  t.integer  "rel_object_id"
  t.text     "object_name"
  t.string   "rel_object_type"
  t.datetime "created_on",
  t.integer  "created_by_id",
  t.boolean  "is_private",
  t.boolean  "is_silent",
  t.integer  "action_id",
  t.integer  "page_id",
end

In my case, most of my objects were linked to pages. And for those, I only wanted page activity to be listed once. e.g. “Modified page X” instead of “Modified object 1 on page X”, “Modified object 2 on page X”.

So I needed to group by page_id, created_by_id, created_on (as a date), and both rel_object_id and rel_object_id.

First I came up with the following :group :

"created_by_id, 
 date(created_on), 
 page_id, 
 rel_object_type || rel_object_id"

Note: the “||” operator in SQLite and PostgreSQL concatenates strings.

Unfortunately that doesn’t work properly since I have other objects which aren’t linked to a page, and they would only get listed once (since for all of them page_id would be NULL). So I needed to use the “CASE” statement to differentiate between the two:

"created_by_id, 
 date(created_on), 
 CASE page_id ISNULL 
   WHEN 1 THEN rel_object_type || rel_object_id 
   ELSE page_id 
 END"

So now both the page objects and the regular objects were listed once per day. But there was another problem.

After the query I used the group_by method to group everything into blocks based on the date. But I also use the time zone support in Rails 2.1, and since the database stores its dates in UTC, I got this rather odd issue where in certain circumstances objects were listed twice.

In reality they were the same date, in UTC. But not in the current timezone I was using. So after a bit of investigation I came up with a solution. I needed to offset the date in the query by the UTC offset for the current timezone.

It turns out that there are a ton of different ways to do it, depending on which database you are using. In my case I was testing with SQLite, so the following sufficed:

date(created_on, '+#{Time.zone.utc_offset} seconds')

And for MYSQL (and perhaps others), using INTERVAL works:

date(created_on + INTERVAL #{Time.zone.utc_offset} SECOND)

Of course, this still has its problems. Like what about daylight savings time?

Unfortunately, since there doesn’t seem to be any set standard for specifying what timezone to evaluate times in, you are either going to have to write a specific case for it, or just put up with the dates potentially being off for an hour or two.

The end result

Well, it looks something like this:

bookmark_borderCloning Backpack

Just over a month ago now, I got sick of my chief open source project, RailsCollab. Nobody seemed interested in it, and the code was growing tired and deprecated. So I decided to try making another open Ruby on Rails project.

Like my previous chief open source project, I decided my new one was also going to be a clone. Mainly because I couldn’t really come up with anything else more interesting, bu I also wanted to see what it would be like cloning something from scratch.

After much deliberation, I decided the focus of my attention would be to clone another of 37Signals’ products. Not Basecamp this time – that has already been done to death. Instead, I decided to try and clone… Backpack!

Interestingly while I looked far and wide for a comparable product, I couldn’t seem to find anything. The only one that came remotely close was TiddlyWiki. But while that was free and open, I felt it completely missed the niche that Backpack filled. Collaboratively edited pages, which are actually easy to edit.

The nitty gritty

First of all – before I even writ one line of code – I decided to check out Backpack’s underlying data models. They told me a lot about how Backpack worked, and of course they gave me a few hints about what data I should store.

Backpack

Models

(generally created and modified dates exist for all of these) 

Pages
    - scope
    - title
    - email_address

    Belongings (basically reference from page -> list item + position.
                     referred to by id)
    Lists
        - name
        List Items
            - completed
            - content
            - order
    Notes
        - title
        - content
    Seperators
        - name
        - page id
    Tags
        - name
    WriteboardLink
Writeboard
    - title

    WriteboardPages
        - content
Reminders
    - at time
    - content
    - creator
    - remindees [Users]
Emails
    - title
    - content
Statuses (one per user)
    - message (text)
    - last update
    - owner
Journals
    - body (length=255)
    - owner
    - last update
Users
    - name
Calendar
    - color
    - subscription url (pulling from other calendar?)
    - name
    - token (used in url for sharing)

    Events
        - title
        - occurs_at
        - occurs_until
        - reminded_at
        - remind (bool, note fixed time before)
        - all day (bool)

Next I decided to sketch out some of the models. has_many, belongs_to, etc. I also decided to give some models such as the “Belongings” different names, as I thought the originals sucked.

(Note that by this time I decided to concentrate my efforts on making a Page editor, as I felt that was the key defining feature)

After I got that out the way, I built a simple page RESTful scaffold for editing pages. All it did was render page slots and the widgets contained therein. What it lacked was the rather crucial AJAX editing that was prevalent in Backpack.

So then I decided to take a closer look at how Backpack’s AJAX page editing worked. It turned out that there were a few crucial components:

  • The widgets – the real content of the page. e.g. Notes, Lists, Dividers
  • The insertion bar – this pops up when you hover your mouse over the top or bottom of the widgets.
  • The action bar – this pops up when you click “Insert Here” on the insertion bar, and is inserted between the relevant widgets.
  • The actions on the action bar – when you click on them, either a widget is inserted (e.g. list) or a form is shown in the slate.
  • The slate – this either exists in a container before the widgets, or in the action bar. It contains most of the forms.

These components are almost exclusively powered by client-side Javascript. Only when you get to the parts where content is added, edited, or removed from a page do you get an AJAX request sent to the server. e.g. Inserting edit forms, adding widgets and belongings, etc.

What you get back from the server is usually Javascript code which alters the content in the page according to the request.

So once I figured all that out, I eventually managed to implement a rather nifty AJAX-powered page editor which somewhat resembled Backpack’s. Mission accomplished!

The result

Well, here it is. It’s called Rucksack, and it acts similarly to Backpack. You can make pages, and share them with others to edit collaboratively. Strange but true.

“Now wait a minute!” you might be thinking, “it looks like crap, and it doesn’t implement everything”. True, but remember that I was only interested in implementing a proof-of-concept page editor. I wasn’t planning on implementing everything and making it look nice. That is for later.

Now while I did start off thinking of Rucksack as a Backpack clone, I don’t think it is the best frame of mind to be in. So for future development, I will be concentrating less on copying backpack, and more on tailoring it to what I think will be useful in organising.

Meanwhile, I have put the code up on github for anyone to check out.

bookmark_borderJavaScript 2 dead? Have no fear, haXe is here!

Earlier on, I noticed this little gem on reddit:

Which led me to this rather interesting mailing list post by Douglas Crockford:

I apologize for my ultrablunt comment. Let me be clear about what I 
meant. It was decided at the Oslo meeting that the project formerly 
known as ES4 is no more. Instead, there will be a new project, 
Harmony, which will be the work of a unified working group. We have 
not yet agreed on a set of goals for Harmony.

I recommend that we close the ES4 list and open a harmony list to 
support this new effort.

Looks like nobody can agree on any set goals for JavaScript 2 (i.e. ES4). Which leads me to think that as with a lot of web standards, JavaScript 2 is just a pipe dream.

Fortunately though, the real world has a solution which already works. Its called haXe, and it implements a lot of what JavaScript 2 should be.

So instead of waiting about for some standards crackpots trying to cobble together a standard, you can write some real code.

Have fun with haXe.

bookmark_borderFuzzy Times

Recently I have been writing a simple web-based reminder app which requires one to input dates and times. To input the date and time, normally one would add some sort of calendar widget which pops up.

But personally I think this tends to be really awkward, especially if you don’t want to be specific about dates or times. e.g. Speculative Opportunities. It also requires a lot of mouse clicking to find and enter the correct date and time.

Fortunately, there are libraries about which aim to solve this issue by allowing you to specify the time in English. Typically they are referred to as “Natural language” date/time parsers. In my case, I found one called Chronic which is distributed as a gem for Ruby. It has even got a nice screencast.

sudo gem install chronic

Consisting of only one public function, Chronic is really easy to use:

require 'chronic'
Chronic.parse("tomorrow at 5")
#=> Fri Aug 08 17:00:00 +0100 2008

See? Nice and simple. We can also use the options to help chronic get the right time. e.g. If I really meant to say “Tomorrow at 5 AM”, I could fix it like this:

require 'chronic'
Chronic.parse("tomorrow at 5", :ambiguous_time_range => :none)
#=> Fri Aug 08 5:00:00 +0100 2008

The options you can choose from are as follows:

  • :context – the context in which the time is assessed (__:past__ or :future).
  • :now – current time.
  • :guess – if false, this returns a time range instead of guessing at a specific time.
  • :ambiguous_time_range – range in hours in which an ambiguous time will be resolved. Best to think of it as hours to skip in the day when picking a time. e.g. setting this to 18 and inputting 5 will result in 5am the next day being chosen.

Sadly I couldn’t seem to find any way of setting which time zone to evaluate the time in. This would have been useful when working with Ruby on Rails 2.1’s new TimeZone support. Fortunately though, I figured out a workaround which is as follows:

# grab time using current time zone as reference
ctime = Chronic.parse(value, :now => Time.zone.now)
# re-interpret time in current timezone
ctime = Time.zone.local(ctime.year, ctime.mon, ctime.day, ctime.hour, ctime.min, ctime.sec)

Basically this gives Chronic the time in the current time zone, which deals with relative times (e.g. “tomorrow”). It also re-interprets the calculated time in case you are a bit more specific (e.g. “5AM”).

For something a bit less hackish, one might want to check out technoweenie’s fork on github. This appears to allow you to tell Chronic to use a different Time class for calculating times, which should solve the problem.

So to conclude, I think Chronic is a nice and simple solution that works rather nicely for common cases.

Other libraries

If you aren’t using Ruby, have no fear. There are similar “Natural language” parsers available for other programming environments which work in a similar fashion to Chronic.

bookmark_borderBrain-Dead Canvas Quirk

The HTML5 Canvas. Great, isn’t it? You can draw pretty much anything you want in a pre-defined area. In more recent browsers, you can even get & set pixel data too, so you pretty much have full control over how things look.

The problem

Unfortunately, the Canvas suffers from a pretty brain-dead design flaw. That is while you can get and set pixels, you cannot control when the pixel data you grab with getPixelData() will be freed from memory. This is exacerbated by the fact that the whole interface runs on top of JavaScript.

“Now wait a minute”, you might think “doesn’t JavaScript have garbage collection for this sort of stuff?”. Well, yes. The trouble is, JavaScript has no standardized garbage collection system, so depending on which browser you happen to be using it’s pot luck whether or not your pixel data will get freed on time.

An example

To start off with, here’s a really simple example test case, using putImageData(getImageData()).

Now lets use a real example here. A while back, you might recall I was writing a SCUMM interpreter in haXe that just so happened to compile to JavaScript as well. For the SCUMM runtime, I am required to display room graphics overlaid with various objects. The graphics are 8bit and use a palette, and that palette can be changed at runtime.

R - PALETTE - RGB

For performance reasons, I decode the graphics and store them in Canvas elements using only the red channel. In fact, throughout the graphics processing pipeline, the red channel is the only part of the pixel data I use. I also make heavy use of the native blitting functions (e.g. fillRect).

It’s not until I come to display the graphics that I re-map the pixels from the red channel into the final colours using the current palette. This requires me to getImageData() the current pixels in the canvas, iterate through and set the data, and then putImageData() back to the Canvas for the final result.

The trouble is since I am re-mapping the pixels every frame, memory usage on certain browsers (Opera, Firefox) skyrockets up to a certain threshold – anything up to 2gb depending on what is happening.

For the test case running on OS X, Opera> tops out at ~600mb. Webkit and Firefox both top out at around 1.25gb (honestly though, I just stopped looking as it was getting stupid). It’s not until I either suspend re-draw and wait several seconds, or simply close the browser window that memory usage returns to normal.

Oddly enough when re-mapping the pixels in my SCUMM interpreter (not in the test case), Webkit’s memory usage stays constant. This flies in the face of what one would expect to happen based on the test case. As for why, it could be anything. The smaller canvas size, assignment of variables – who knows.

Workarounds?

Realistic workarounds include:

  • Perform all graphics operations manually on a single set of pixel data (too slow!)
  • Use drawImage() as opposed to getImageData() & putImageData()
  • Limit the amount of screen updates
  • When working with palletised image data that needs re-mapping, store it as RGB, and re-calculate it only when the palette changes
  • Don’t use getImageData() at all, especially considering it doesn’t seem to work in Safari at the moment.

Either that, or simply implement everything which needs pixel-level graphics manipulation in Flash or a Java Applet, foregoing the pure Javascript ethic.