Documentation

After taking a long break over the winter I've started working on neutrino again. When I left off I was working on generating static data in the binary object files: basically, taking neutrino objects and constructing binary data in the generated executables. I know how my code generator should behave for this to work but changing it to do that has been tricky because it is a large amount of relatively complex code (generating binaries is a relatively complex process) and I've forgotten how it works exactly. The problem is that I didn't do a good job of documenting it. So my step one is going to be documenting it. Or, it turns out, that actually will be step two because at this point neutrino doesn't have a documentation format - so step one will be implementing one.

JavaDoc
I've always felt that java's JavaDoc comments were a good idea in principle but problematic in a lot of the particulars. The good part is that, unlike systems like doxygen and JsDoc, it's a standard that comes with the language, and it provides all the primitives I want then documenting code. However...

JavaDoc builds on top of block comments. Block comments are for free-form, anything goes text. This means that it's tricky if the language wants to impose rules on them, even if those rules might be useful. For instance, it's legal to @link to a class or method that doesn't exist.

Also, since they it is based on comments the connection between JavaDoc and the code it documents is weak. This, for instance, is meaningless but perfectly legal:

String x = /** Looky here! */ "foo";

Finally, JavaDoc builds directly on HTML which is fine if what you want to generate is HTML but problematic if you want to generate anything else. It is also difficult to read outside of a browser and I almost always read the documentation in the code, not in the generated output.

A small detail that's always bugged me about them, also, is that they waste two lines of space:

/**
 * Two lines of comment takes up three lines of code
 * because of the start and end markers.
 */

This is a two-line comment, really, and it shouldn't take up more than two lines in the program.

For all these reasons I'm not keen to base neutrino's documentation format on JavaDoc and I'm not familiar with any other formats that solve all of these issue. So I've ended up starting basically from scratch, with some inspiration from a language I previously worked on, neptune.

Neutrino
I'll start with the two last points first, the syntax. Here's one example of what a piece of documentation looks like:

/| Moves the specified number of disks from the @from@
 | peg to the @to@ peg.
 |
 | param(from): Which peg to move from.
 | param(to): Which peg to move to.
\| param(disks): How many disks to move.
def this.move(from:, to:, disks:) {
  ...
}

I'd like the syntax to be such that you don't need the start and end marker to require a separate line each. In other words, you need to be able to write the documentation on the same lines as those markers. So the way this works is: /| starts a comment and \| marks that that lines is the end of the block, but the whole line is included in the comment. These last markers can also occur alone, without a start marker, in which case it's a single-line doc comment:

\| Creates a new hanoi instance.
@static def Hanoi.new() => ...

The first character on each line (above that would be the | character) are used to determine how to interpret each line. A | means that it is just text. A @ means that the text is code. For instance:

/| Moves the specified number of disks from the @from@
 | peg to the @to@ peg. For instance, to execute the
 | towers of hanoi with 15 disks do:
 |
 @ def hanoi := new Hanoi();
 @ hanoi.move(from: 0, to: 2, disks: 15);
 |
 | param(from): Which peg to move from.
 | param(to): Which peg to move to.
\| param(disks): How many disks to move.
def this.move(from:, to:, disks:) {
  ...
}

This way it's as easy to identify and read code examples within the code as it is for a doc generator to recognize.

The second thing that makes these different from JavaDoc is that they're not comments, they're annotations and can only occur in the same positions as annotations. Documentation always belongs to a particular element and it is clear from the grammar which element that is. The code above is, in principle, equivalent to:

@doc(...) def this.move(from:, to:, disks:) {
  ...
}

This also means that documentation can stay attached to source elements at runtime and be accessed through reflection, like any other annotations.

Finally, the format used is not HTML but something very close to textile. This is, again, to make it easier to read from source code, and to make it easier to generate documentation in other formats than HTML. I remember we had a doc generator for neptune that generated a PDF file for a module, which worked really well (and looked, if I do say so myself, awesome).

I think this format has a lot of appealing properties and I'm actually eager to get started documenting the code generator.

Neutrino

In the past I've written on this blog about various programming language projects I've been working on, including neptune and saturn. It's been a while though, mainly because I've been working on v8 and spent more than enough time implementing languages at work so I didn't have the energy to do it as a hobby as well.

That is no longer the case and I am indeed back on the horse working on the next language, neutrino. I'm trying a different approach to writing about it this time since there are a lot of interesting implementation issues as well as language design issues. I've started a new blog, the neutrino blog, where I'll post about language design and implementation progress. Go on, take a look!

Scribble

I recently came across an old note pad I had been using about two years ago. I like to keep all my old note pads; they're like raw dumps of what I was doing then and the ideas I was playing around with. This particular one was the one I was using while reading the World's Writing Systems, the mother lode for anyone interested in writing systems.

The image on the right is one of the pages. After reading about so many different writing systems I just had to try making my own. The underlying language is English but unlike the latin system it has only a cursive form and makes heavy use of diacritics, which makes it a lot more compact and altogether just a very different kind of system to use.

Actually, what you see is the system in its most primitive and childish rendering. It's very geometric and angular, like something you might see on an alien spaceship in a bad science fiction movie. That's not exactly the look I was going for. But then I had only just learned it when I wrote this. My idea was to actually learn to use it, learn which corners could be cut without losing readability, evolve some comfortable ligatures, basically the same thing I did to get from the handwriting I had when I first learned the latin alphabet to my current utilitarian but definitely more aesthetically pleasing hand. That takes a long time but if I don't lose interest before then I'll be back in 10-15 years to present a much nicer-looking cursive handwriting.

Scsh

I use shell scripts a lot. To automate things I do often. To tie together different commands like grep and sed. For one-shot tasks like running the same command 100 times and calculate the average execution time. Basically, when want to instruct my machine to carry out some operation I'll almost always do it by invoking a shell script, usually one written by myself.

Up until I left the v8 project I'd written most shell scripts in bash or, if that became too horrible, python. But when I left v8 to work on wave I made a decision: no more. I don't want to be stuck with a choice between a language that is, frankly, grotesquely horrible, bash, or one that is okay but just not made for what I was using it for, python.

That's when I remembered scsh, the scheme shell (pronounced "skish", rhymes with fish). I had read Olin Shivers' report on it and of course the famous acknowledgements and I'd always thought it was a brilliant idea to use scheme as a glue language. I'd never actually tried the tool though so I decided that this was the perfect time to give it a try.

The latest release of scsh is from 2006. You don't get the impression that it's a project under active development. It's also not available on most system I use. This would normally have put me off but the though that it might rid my life of bash motivated me to give it a try anyway.

On mac it's easy to install, just

sudo port install scsh

On my linux machine I had to build and install it myself, and I had to tweak the build files a little for it to build on a 64 bit machine; if I remember correctly all I had to do was add -m32 at the right place in a generated Makefile.

Having installed it I was ready to start running shell scripts written in scheme. Or so I thought. I wrote my first script,

#!/usr/local/bin/scsh

(display "Hello World!")

and ran it. No dice.

$ ./helloworld.ss
Unknown switch ./helloworld.ss 
Usage: scsh [meta-arg] [switch ..] [end-option arg ...]

meta-arg: \ 

switch:
        ... snip ...
        -s  <script>   Specify script.
        ... snip ...

Ah, I forgot to use the -s option. Add that, try again:

$ ./helloworld.ss
Error: EOF inside block comment -- #! missing a closing !#
       #{Input-port #{Input-channel "./helloworld.ss"}}

Okay, now it's running the script in scsh but it chokes on the #!. For a language designed to run shell scripts it's surprisingly uncooperative. After experimenting a while and a few google searches I came upon the required magic enchantment. My script was now:

#!/usr/local/bin/scsh -s
!#

(display "Hello World!")

While this violates POLA in 100 different ways it works. Yay! The reason it works is because #! ... !# happens to be the block comment syntax in scheme, or at least scheme48 which is the implementation scsh is based on.

Okay, now I could start actually using it to write scripts. The first thing I wanted to implement was a set of wrappers that helped keep track of a handful of git clones of the same underlying non-git repository. That way I can keep tests running and build output intact in one workspace while I work on something else in another separate one, something that using different git branches in the same workspace doesn't give you.

Scsh uses macros and unquote to run external commands so for instance this function,

(define (git-new-branch name)
  (run (git checkout -b ,name))

will run the command

git checkout -b <name>

The , means that the value of the parameter name should be inserted there. The output goes to standard output. You can also get the output back as a list of strings by using run/strings instead of run. As an example of using that here's a function that returns whether or not a git repository has pending changes:

(define CHANGE-RX
  (rx (| "Changed but not updated:" "Changes to be committed:")))
(define (git-has-changes)
  (call-with-current-continuation
    (lambda (return)
      (define (process-line line)
 (if (string-match CHANGE-RX line)
     (return #t)))
      (let ((output (run/strings (git status))))
 (map process-line output))
 (return #f))))

This code runs git status and then iterates through the strings returned looking for the string Changed but not updated: and Changes to be committed:, returning immediately when it finds one. Scsh comes with a rich regular expression library which is the rx part above. It's more verbose than POSIX regexps and does lack some of the conveniences, but is on the other hand much more straightforward and readable and seems to be at least as, if not more, powerful.

At this point you may say: hey, I could have written that function in one line using grep. And you could. The difference is that when I use grep the complexity of my script increases exponentially with the complexity of what I'm trying to accomplish. With scsh a script may start out a bit more verbose, as above, but when the script grows a little more complex, as they tend to do, I can solve the problem using standard high-level programming constructs that are already hardwired in my brain instead of having to pore over the grep manpage to figure out how I make it do what I'm trying to do.

For instance, I can write a one-line script using find and sed that removes all lines containing a.b.c.X from a file, easy. But if I want to extend my script a bit so it only removes a.b.c.X when it occurs within a block of lines enclosed in square brackets that also contains a.b.c.Y the problem has become too complex for me to solve by chaining together shell commands. I'm sure it can be done but I would have to spend an inordinate amount of time figuring out how. On the other hand, doing this in scsh I can solve each individual problem separately: finding blocks enclosed in square brackets, searching for a.b.c.Y, deleting lines containing a.b.c.X, and and combine the individual operations using standard language constructs.

;; This script removes all lines enclosed in brackets containing
;; |to-be-removed| but only if the block also contains
;; |removal-indicator|.
(define (main args)
  (let ((to-be-removed (cadr args))
        (removal-indicator (caddr args))
        (file-name (cadddr args)))
    ;; Regexp matching python lists
    (define LIST-RE
      (rx (: "[" (submatch (* (~ "]"))) "]")))
    ;; Regexp matching lines containing |to-be-removed|.
    (define STRIP-REMOVED-RE
      (rx
        (: #\newline 
           (* (~ #\newline)) 
           ,to-be-removed
           (* (~ #\newline)))))
    ;; Processes all matches
    (define (process-input str)
      (regexp-substitute/global
        #f LIST-RE str 'pre process-list 'post))
    ;; Processes the contents of square brackets
    (define (process-list match)
      (let ((result-contents (remove-if-required
                               (match:substring match 1))))
        (string-append "[" result-contents "]")))
    ;; Removes all occurrences of |to-be-removed| where it is
    ;; together with |removal-indicator|
    (define (remove-if-required str)
      (if (and
            (string-contains str removal-indicator)
            (string-contains str to-be-removed))
          (strip-removed str)
          str))
    ;; Removes one line containing |to-be-removed|
    (define (strip-removed str)
      (regexp-substitute/global
        #f STRIP-REMOVED-RE str 'pre "" 'post))
    (let*  ((input-port (open-input-file file-name))
      (input (read-string 100000 input-port)))
      (display (process-input input) (open-output-file file-name)))))

This is more verbose but took a lot less time to write and debug than it would have taken me to write an equivalent bash script, and it will be much easier to understand and extend later on. And this is scsh competing with bash where bash is strong. As the complexity of your script increases the power of scsh's abstractions becomes more and more apparent. Here's the command I use to update all my git clones from the central repository, using one über-workspace that stays in sync with the underlying repository and then a number of unter-workspaces that clone the über-workspace:

;; Performs the work of a 'sync' operation.
(define (run-sync)
  (within (@workspace UBER-WORKSPACE)
    (within (@git-branch MASTER-BRANCH-NAME)
      (sync-from-repository)))
  (for (workspace in UNTER-WORKSPACES)
    (within (@workspace workspace)
      (pull-from-master workspace))))

This is not pseudo-code, this is literally the code that is run. The within form takes care of entering something, a directory, branch or whatever, carrying out an operation and leaving again, dealing gracefully with errors at any point. The sync-from-repository and pull-from-master functions are straightforward one-line calls to external tools. I usually wrap external calls in a function that includes logging to make debugging easier.

The above function uses a number of generally useful abstractions, including the within and for forms which, it should be noted, are not built into scsh, I defined those myself using scheme's define-syntax. You would obviously like these abstractions to live in separate files that could be shared between different scripts. Importing or including other files is not one of scsh's strong sides. There is a module system that I have no doubt is powerful and clever, but I just didn't have the patience to figure out how it worked so I use a scsh runner script that takes care of loading libraries before starting your script:

#!/bin/sh
#
# Usage: scsh.sh <scsh script> <options> ...
#
# Loads the specified scsh script and all .sm files in the same
# directory and calls the 'main' function.

SCRIPT=$1
ROOT=`dirname $SCRIPT`
LIBS=`ls $ROOT/*.sm | sort | xargs -n1 -i@ echo -l @`
MAIN=main

exec /usr/local/bin/scsh $LIBS -e $MAIN -s $*

Using this script rather than calling scsh directly I can factor utilities out into .sm (scsh module) files and have them loaded automatically. And with a library loading mechanism in place, and a bit of practice with scheme and some basic convenient utilities in place, scsh is an extremely powerful tool. Here are some more examples taking from my scripts.

Here's an example of defining command-line options

(define parse-options
  (option-parser
    ((--runs r)
      (set! number-of-runs r))
     (--help
      (exit-with-usage))))

The option-parser form lets you define a number of command-line options and the action to perform when the option is encountered. It returns a function that performs the appropriate processing and returns a list of those arguments that were left when removing all the ones that were recognized.

This command enters the branch in the über-workspace that corresponds to your current git branch in an unter-workspace and asks for the underlying changelist id:

(define (get-current-cl)
  (let* ((all-branches (git-list-branches))
         (current-branch (car all-branches)))
    (within (@review-client current-branch)
      (get-current-cl))))

Again, this command actually does a lot of work but you hardly notice because it's all been packed away in the various abstractions. You only see the high-level structure of what's going on.

This command pushes the current unter-workspace branch to the über-workspace, enters that branch and exports it as a change against the underlying repository:

(define (run-export)
  (let* ((all-branches (git-list-branches))
         (current-branch (car all-branches)))
    (git-push "origin" current-branch)
    (within (@workspace UBER-WORKSPACE)
      (within (@git-branch current-branch)
        (export-change)))))

Overall I'd suggest that if you've ever been frustrated with traditional shell scripts and have a basic knowledge of scheme, or want to learn it, you should give scsh a chance. I've never been a lisp or scheme fanatic but despite some amount of unfriendliness from the tool itself, including the odd way you have to invoke it and poor error reporting, I've been totally won over. Scsh FTW!

Sydney

Wow, it's been a long time since I've posted here. I've actually had a lot of material for blogposts, most recently about my adventures with scsh, but it somehow doesn't manage to get from my head onto the blog.

So, what's new with me? Well...

A few months ago I left the v8 project which I had worked on almost since it started. I've enjoyed working on v8 a lot and it's been a great experience to be an active part in the browser race over the last few years. However, I always knew that I wasn't going to spend the rest of my life implementing JavaScript.

Last year I visited Sydney and spent a few weeks working at Google's office there. I happened to be there during the first big demo of Google Wave and watched it with the team which is mainly located in Sydney. I was totally blown away, and that feeling never really wore off. It's now one year and quite a bit of paperwork later; I'm now living in Sydney and working on wave.

Wave is a very different kind of project than v8 and a departure from the programming language related projects I've worked on so far. That's a feature, not a bug.

Of course, now that I'm drinking the wave Kool-Aid I simply must include a wave element for comments.

Double

Many (most?) virtual machines for dynamically typed languages use tagged integers. On a 32-bit system the "natural" tagsize for a value is 2 bits since objects are (or can easily be) 4-byte aligned. This gives you the ability to distinguish between four different types of values each with 30 bits of data, or three different types where two have 30 bits and one has 31 bits of data. V8 uses the latter model: 31 bit integers, 30 bit object pointers and 30 bit "failure" pointers used for internal bookkeeping. On 64-bit systems the natural tagsize is 3 which allows for more types of tagged values, and inspired by this I've been playing around with tagging another kind of value: double precision floating-point numbers.

Whenever you want to tag a value you need to find space in the value for the tag. With objects it's easy: they're already aligned so the lowest few bits are automatically free:

oooooooooooooo..oooooooooooooo000 (untagged)
oooooooooooooo..oooooooooooooottt (tagged)

With integers no bits are free automatically so you only tag if the highest bits are the same, by shifting the lowest bits up and adding the tag in the lowest bits, for instance:

0000iiiiiiiiiiiii..iiiiiiiiiiiiii (untagged)
0iiiiiiiiiiiii..iiiiiiiiiiiiiittt (tagged)

The reason both the highest bits 0000 and 1111 work is that when you shift up by 3 there will be one bit of the original 4 left, and when shifting down again sign extension spreads that bit back out over all the top 4 bits.

With doubles it becomes harder to find free bits. A double is represented as one sign bit s, an 11-bit exponent e and a 52 bit fraction, f:

seeeeeeeeeeeffffffffff..fffffffff (untagged)

The value of the number is (-1)^s2^e1.f.

To get a feel for how this representation works I tried decomposing some different values into into their parts, here using single precision but the same pattern applies to double precision:

value	sign	exponent	fraction
1.0	0	0	1
-1.0	1	0	1
2.0	0	1	1
0.5	0	-1	1
3.0	0	1	1.5
100.0	0	6	1.5625
10000.0	0	13	1.2207031
-10000.0	1	13	1.2207031
0.0001	0	-14	1.6384
0	0	-127	0.0 (denormalized)
NaN	0	128	1.5
Infinity	0	128	1.0

Looking at this table it's clear that in an interval around ±1.0, starting close to 0.0 on the one side and stretching far out into the large numbers, the exponent stays fairly small. There are 11 bits available to the exponent, that's a range from -1022 to 1023 since one exponent is used for special values, but for all values between 0.0001 and 10000 for instance its actual value only runs from -14 to 13.

Even though the high bits of the exponent are unused for many numbers you can't just grab them for the tag. The 11 exponent bits don't contain the exponent directly but its value plus a bias of 1023, apparently to make comparison of doubles easier. But after subtracting the bias from the exponent it is indeed just a matter of stealing its top three bits of the value (leaving the sign bit in place):

su000uuuuuuuffffffffff..fffffffff (untagged, bias subtracted)
suuuuuuuuffffffffff..fffffffffttt (tagged, bias subtracted)

Using this approach all doubles whose exponent is between -127 and 128 can be tagged. Since single precision numbers use 8 bits for the exponent all numbers between the numerically greatest and numerically smallest single precision numbers, positive and negative, can be represented as tagged doubles. One potential concern is that this encoding does not allow ±0.0, NaNs or ±∞ but I'm not too worried about that; it's easy to handle 0.0 specially and it's unclear how problematic the other values are.

What's the performance of this? The short answer is: I don't know, haven't tried it. The long answer is: I can guess and I'm fairly certain that it's worth it.

One reason it might not be worth it is if most doubles are not covered by this. To test this I instrumented v8 to test each time a heap number was allocated whether the value would fit as a tagged double. Furthermore I disabled some optimizations that avoid number allocation to make sure the allocation routines saw as many numbers as possible. The results after running the v8 benchmarks were encouraging:

+----------------------------------------+-------------+
| Name                                   | Value       |
+----------------------------------------+-------------+
| c:SmallDoubleZero                      |       16149 |
| c:SmallDoubleNaN                       |           2 |
| c:SmallDoubleInfinity                  |       32959 |
| c:SmallDoubleOther                     |        1290 |
| c:SmallDoubleHits                      |      628660 |

The number in c:SmallDoubleHits is the ones that fit in a tagged double: 92%. Furthermore, of the numbers that could not be represented all but 2% were zero or infinity. The v8 benchmark suite contains two floating-point heavy benchmarks: a cryptography benchmark and a raytracer. Interestingly, the cryptography benchmarks is responsible for almost all the ∞s and the raytracer is responsible for almost all the 0.0s. If a special case was added for 0.0 then 99.3% of all the doubles used in the raytracer would not require heap allocation.

Note: people often ask if we plan to add 64-bit support to v8. Don't read this as an indication one way or the other. I'm just testing this using v8 because that's the code base and benchmarks I know best.

Note that this is just a single data point, there's no basis to conclude that the same holds across most applications. Note also that since v8 uses tagged integers there is some overlap between this an existing optimizations. But since all 32-bit integers can be represented as tagged doubles any nonzero numbers that are not counted because they were recognized as small integers would have counted as tagged doubles.

Another reason it might not be worth it could be that tagging and untagging is expensive; I doubt that it is considering how relatively few operations it takes:

static const uint64_t kBias = 0x3800000000000000ULL;
static const uint64_t kStolenBitsMask = 0x7000000000000000ULL;
static const uint64_t kLowerMask = 0xfffffffffffffffULL;
static const uint64_t kUpperMask = 0x8000000000000000ULL;
static const uint64_t kTagSize = 3;
static const uint64_t kTag = 0x3;

uint64_t double_bits(double value) {
  return *reinterpret_cast<uint64_t*>(&value);
}

double make_double(uint64_t value) {
  return *reinterpret_cast<double*>(&value);
}

bool fits_small_double(double value) {
  return ((double_bits(value) - kBias) & kStolenBitsMask) == 0;
}

uint64_t tag_small_double(double value) {
  uint64_t bits = double_bits(value) - kBias;
  return (bits & kUpperMask)
       | ((bits & kLowerMask) << kTagSize)
       | kTag;
}

double untag_small_double(uint64_t bits) {
  uint64_t result = (bits & kUpperMask)
                  | ((bits >> kTagSize) & kLowerMask);
  return make_double(result + kBias);
}

I haven't benchmarked this so I can only guess about the performance. I have no doubt that testing and tagging a double is faster overall than allocating a heap object but I suspect reading from memory could turn out to be cheaper than untagging. The overall performance also depends on how doubles are used: if most are only used briefly and then discarded then the possible cheaper reads don't make up for the more expensive allocation. But I'm just guessing here.

In any case I think this is a promising strawman and it would be interesting to see how it performed in a real application. I also suspect there are more efficient ways of tagging/untagging the same class of doubles. The problem is actually a fairly simple one: reduce the highest bits from 01000, 11000, 01111 or 11111 to only two bits and be able to go back again to the same five bits.

Further reading: alternative approaches are here, which is essentially the same as this but has more special cases, and this which goes the other way and stores other values in the payload of NaN values.

Finally

Before I had used C++ I had quite a strong dislike for it. I remember wondering why anyone would voluntarily use C++? and being frustrated when reading interviews with Bjarne Stroustrup because he wouldn't admit that C++, or at least parts of it, sucks.

Well, for the last few years I've actually been using C++ and have ended up changing my mind completely. As I expect most C++ programmers do I stay within a subset of the language, but that subset is now one of my favorite languages: not too complicated, powerful and concise. With the other languages I use regularly, Java, Python and JavaScript, I often find myself thinking "man, what were they thinking when they designed this feature?". That practically never happens with C++. It can be complex and confusing but when taking the constraints and interactions with other language features into account I can't think of a single instance where I've been able to come up with a better solution. I realize that is quite an arrogant way to put it but look at it this way: coming from an arrogant and opinionated person it is a huge compliment.

One of my favorite features is stack-allocated objects because they enable the resource acquisition is initialization pattern. Given a choice between doing resource handling in Java, using finalizers for objects and finally clauses for scopes, and C++ using RAII, I would choose C++ any day.

The weaknesses of finalizers are well-known so I won't say more about them. The problem with finally clauses is closely related to the subject of making wrong code look wrong. Finally clauses divorce the code that releases a resource from the code that allocates it, which makes it much harder to see when you forget to clean up.

FileReader in = new FileReader(fileName);
try {
  /* process file */
} finally {
  in.close();
}

Using a finally clause means having the code for cleaning up in on the other side of the code that uses it. You have to scan the whole method to convince yourself that in will indeed be closed. An even worse issue is the fact that the try part has a scope of its own that the finally clause can't see. This means that anything you do after having allocated a resource will, by default, be hidden from the finally clause. Because of this the pattern above doesn't scale if you want to open two files:

FileReader in = new FileReader(inFileName);
FileWriter out = new FileWriter(outFileName);
try {
  /* process files */
} finally {
  in.close();
  out.close();
}

If the second constructor throws an exception, and you can't be sure it doesn't, then the finally clause is not yet in effect and so in won't be cleaned up. You can't declare the out variable in the try clause because then the finally clause can't see it so what people often end up doing is:

FileReader in = null;
FileWriter out = null;
try {
  in = new FileReader(inFileName);
  out = new FileWriter(outFileName);
  /* process files */
} finally {
  if (in != null) in.close();
  if (out != null) out.close();[1]
}

What we're trying to do here is really very simple: we want to be sure that these files are closed before leaving the scope. With try/finally the solution is more complicated than the problem.

In C++ the destructor of stack-allocated objects give you a way to execute code exactly when you leave a scope. Here's how you could express the same thing in C++:

own_resource<FILE*> in_resource = fopen(in_file_name, "r");
FILE* in = *in_resource;
if (in == NULL) return;
own_resource<FILE*> out_resource = fopen(out_file_name, "w");
FILE* out = *out_resource;
if (out == NULL) return;
/* process files */

The resources we acquire here are stored in a stack-allocated own_resource; the destructor of the own_resource class takes care of releasing the value stored in it. The language ensures that no matter how control leaves the current scope the own_resource destructors will be called first. Furthermore I can see right at the call to fopen that the result will be cleaned up. You don't have to use this pattern long before any call that allocates a resource and does not store it in an own_resource (or an own_ptr or some other own_ object) looks very wrong.

How to manage resources is closely related to how to signal errors. Most new languages use some form of exceptions, even though they have acquired something of a bad name. There are many problems with exceptions (none of which can be solved with error codes by the way) but being able to specify how a resource should be cleaned up at the same time as you allocate it does solve many of those problems. Getting this right requires you to assume that any part of your code could throw an exception. It's a slightly nicer version of preemption really: someone might interrupt you at any nontrivial operation but unlike preemption all you must be able to do is bail out, you don't have to be able to continue running. That is doable. Furthermore it's just a fact of life: whether you use exceptions or not you may have to bail out at any nontrivial operation, because of stack overflow, memory exhaustion, SIGINT, or whatever. The best way to defend yourself against that is to program defensively, and to write your code in a style where resource allocation operations stand out if they don't immediately make provisions for deallocation. I don't know a language that does that better than C++.

However, as much as I've been praising C++, their solution relies much too heavily on static typing for my taste. An interesting challenge is to design a way for flexible object-oriented RAII to work in dynamically typed languages. That is something I will be working on and I hope I'll eventually be back here with a good solution.

---

¹: As Bob points out in the comments there's a bug here: if in.close() throws an exception then out will not be closed. It's another illustration of how hard this stuff is to get right. Note that the C++ example is not affected by this issue: if the destructor for out (which is the first to be called) throws an exception the same rules apply with regards to in: the exception causes us to leave its scope so its destructor will be called.