Tony and me at the moment of truth

Jamie — who knows the term “source code” primarily as the title of the movie Jake Gyllenhaal did after Prince of Persia — digested my explanation; then, looking as confused as before, asked “Why?!?”

It was such a simple question, it stumped me for a moment. Why would I spend a whole day trying to recover data from some ancient floppy disks?

I said: “Because if we didn’t, it might have disappeared forever.”

Why source code?

POP source code recovered after 22 years

You don’t, if all you want is to listen and enjoy the music. But to a pianist performing the piece, or a composer who wants to study it or arrange it for different instruments, the original score is valuable.

It’s possible, up to a point, to reverse-engineer new source code from a published video game, much as a capable musician can transcribe a musical score from listening to a performance. But in both cases, there’s no substitute for the original document as a direct line to the creator’s intentions and work process. As such, it has both practical and historical value, to the small subset of the game-playing/music-listening community that cares.

This is why I was so sorry to have lost the Prince of Persia source code, and happy to find it again.

Lost and found (Geek quotient = 9)

If you’ve read my 1980s game dev journals, you know that by the time Prince of Persia shipped in 1989, I was burned out on coding and seriously eager for the next chapter of my life to start. So I did what most programmers would do: I backed up my Apple II source code onto 3.5″ floppies, stuck it in a box, and promptly forgot about it.

Thirteen years later, when I looked for that box of source code again, I couldn’t find it. I was in Montreal with an amazing team making Prince of Persia: The Sands of Time. Lead programmers Dominic Couture and Claude Langlais had volunteered to port the original POP to the PlayStation 2, and slip it into our new game as an easter egg. (This was their idea of a fun respite from crunch time.) All they needed was the source code. But as much as we searched — from my garage to Broderbund’s archives to Doug Carlston’s basement — it was nowhere to be found.

Finally we tracked down Scott Shumway, who’d done the 1992 Mac port of POP. He didn’t have the Apple II source code either, but he did have the Mac source code. Dom and Claude made short work of porting it to the PS2, and Sands of Time got its easter egg. Everyone was happy.

On my return to LA, I dug deeper, and turned up a whole shoebox full of Apple II floppies, some dating back to high school. The source code to all my early games was in there: Deathbounce, Karateka. But no POP.

I didn’t need the source code for anything; and it wasn’t as if POP had been lost to history — vintage Apple II POP copies (and their disk images) were widely available — but still, it bothered me to think that something I’d spent years working on was just gone. I felt dumb for not having kept a copy.

This was eight years ago. I gave up the search and forgot about it.

Until two weeks ago, when my Dad shipped me a carton of my stuff he’d found cleaning out the closets of his New York apartment. Inside was the source code archive I’d mislaid in 1990.

Paper is forever

Here’s the thing about digital backups and magnetic storage media: They don’t last very long.

Try popping your old 1980s VHS and Hi-8 home movies into a player (if you can find one). Odds are at least some of them will be visibly degraded or downwright unplayable. Digital photos I burned onto DVD or backed up onto Zip disks or external hard drives just ten years ago are hit and miss — assuming I still have the hardware to read them.

Whereas my parents’ Super 8 home movies from the 1960s, and my grandparents’ photos from the 1930s, are still completely usable and will probably remain so fifty years from now.

Pretty much anything on paper or film, if you pop it in a cardboard box and forget about for a few decades, the people of the future will still be able to figure out what it is, or was. Not so with digital media. Operating systems and data formats change every few years, along with the size and shape of the thingy and the thing you need to plug it into. Skip a few updates in a row, and you’re quickly in the territory where special equipment and expertise are needed to recover your data. Add to that the fact that magnetic media degrade with time, a single hard knock or scratch can render a hard drive or floppy disk unreadable, and suddenly the analog media of the past start to look remarkably durable.

This is why, when I posted about finding the POP source code, digital archivist Jason Scott, Apple II collector Tony Diaz, Derek Moore, and the technical teams behind the DiscFerret and Kryoflux disk readers volunteered their time and effort to give us the best possible shot at a successful extraction.

Let me begin to count the ways I’ve been lucky with this: The box was found. The disks were intact. Prince of Persia and I happened to have a high enough public profile that people of Jason and Tony’s caliber (and dozens of others who contributed their expertise via IRC, skype and twitter from around the world) cared.

In the bigger picture, our timing was lucky. The 1980s and the Apple II are long enough ago to be of historical interest, yet recent enough that the people who put the data on the disks are still with us, and young enough to kind of remember how we did it. Roland Gustafsson, author of the special 18-sector RWTS routines that had made our disks super-efficient in 1988 (and unreadable to anyone but us), was able to get on IRC in 2012 and explain what he’d done to Discferret kids who weren’t born then.

For all these lucky reasons, our archaeological expedition was crowned with success.

From a preservationist point of view, the POP source code slipped through a window that is rapidly closing. Anyone who turns up a 1980s disk archive 20 or 30 years from now may be out of luck. Even if it’s something valuable that the world really cares about and is willing to invest time and money into extracting, it will probably be too late.

This is why it’s awesome that there are people out there working on digital preservation. Because now is the time.

Back up your backups

Jason suggests the following rule of thumb: If you have data you want to keep for posterity, follow the Russian doll approach. Back up your old 20GB hard drives into a folder on your new 200GB hard drive. Next year, back up your 200GB hard drive into a folder on your new 1TB hard drive. And so on into the future.

As for me, the past 48 hours have been a fun walk down memory lane. And have given me a renewed appreciation for paper, celluloid, and stone tablets.

(Postscript: For 6502 assembly-language aficionados, the Prince of Persia source code is now up on github, along with a README file that answers some frequently asked questions. I’ve been amazed and moved by the outpouring of interest in the #popsource saga — it literally crashed this website for several hours today.)

Now, I really need to get back to my day job of making up new stuff. I can only hope to have the same lucky, glorious headache of trying to recover some of it 20 years from now.

Further reading:

“Making of Prince of Persia” game development journal

25-year capsule history of Prince of Persia

POP source code on github

Despite my eagerness to see what’s on those disks, I’ve yet to pop them in a drive. As readers of this site have cautioned me, digital media degrade with age; if the disks are in a fragile state, normal handling could damage them further and even render them unreadable.

In today’s guest post, digital archivist Jason Scott explains why reading 20-year-old floppy disks is trickier than it sounds — and why he’s volunteered to fly from NY to LA on Monday with special equipment to tackle the job himself.

Monday will be an exciting day. Much like opening a long-sealed sarcophagus, I truly have no idea whether we’ll find what we’re hoping for, or just data dust. For anyone who wants to share the suspense, we’ll be live-tweeting our progress. Hashtag: #popsource. (I wanted to use #sourcecode, but it was taken!)

Meanwhile, here’s Jason’s story, offering a glimpse behind the scenes of a profession whose existence I couldn’t have foreseen or imagined when I was making Prince of Persia in the 1980s: Digital archeologist.

I first heard about Prince of Persia in a somewhat strange fashion; a high school friend said that David’s older brother was working on a new game to follow up his big hit Karateka. I asked what it was about, and he said it was something about Persian princes and acrobatics. I left it at that, but I knew it’d be great, if Karateka was any indication.

I went to Horace Greeley High School after Jordan, and knew his brother, David, who graduated the same year as me. David was the motion model for Prince of Persia. Jordan was this talented figure somewhere out in the fog of the real world, who was making actual, sold-everywhere games with a company I really liked and respected (Broderbund), and was basically living the dream I hoped to live one day: game developer.

(My own dream was fulfilled — I did work for a short time at Psygnosis, makers of Wipeout, as a tech support phone monkey, and another year stint at a startup game studio, before moving on to other places in the computer world.)

It wasn’t until a couple years ago that I moved away from jobs like system administration and backup-watcher into the world of computer history and documentary filmmaking, where I am now. As one of the Adjunct archivists of the Internet Archive, I seek out new collections of data and help preserve current ones — anything from digitized books and audio to long-forgotten shareware CD-ROMs and obscure information files uploaded years ago. It’s a great time, and most importantly, it affords me the flexibility to travel when I’m needed somewhere.

So this was why, when Jordan announced he’d gotten back the Prince of Persia disks he had in his own collection, a lot of friends of mine started linking me to the article and saying “Well?” It was a perfect fit. I had seen Jordan for a few moments after his recent appearance at GDC, so it made sense to have us talk about my coming in to oversee the retrieval of data from the disks. What a nice journey — from hearing the game was being worked on in my youth to helping make sure Jordan’s work lasts for future generations!

Pulling data off dead media in the present day is both easier than it ever has been, and as frustrating as ever. (When I say “dead,” I mean the format. You can’t really go down to the local store and buy a box of 5.25″ floppy disks any more, nor would you want to — a USB stick will give you well over a million times the space and cost you almost nothing.) Thanks to a lot of work by a lot of different people, pulling the data off these floppies can now be as simple as putting it into a vintage disk drive, or a modified recent one, and pulling the individual sectors right into a file that can go into the internet in seconds. But just as it’s so trivial to do this, any clever tricks done to the floppy that made sense way back then could make it a puzzle wrapped in a goose chase to extract. Not to mention, these discs are old — in this case, at least twenty years old, and they’re just magnetic flaps of plastic sealed inside a couple of other sheets of plastic. A lot can go wrong, and no extraction is guaranteed.

It’s the Friday before I hop into a plane in NY — ironically, just miles from where Jordan’s disks had rested comfortably in the back of a closet for 20 years — to Los Angeles, where he works and lives these days. Once I arrive there, I’ll be joined at the site by someone I reached out and tapped due to his reputation within and outside the Apple II community: Tony Diaz. He’s one of a tireless group of vintage hardware and software collectors working to ensure an entire swath of computing history isn’t lost to the shadows. With a collection of Apple-related hardware that is likely one of the largest in the world, accompanied by attempts to catalog and document as much of it as possible, I knew Tony would be the best partner in this project. Tony will be bringing over a pile of Apple II hardware, maintained and cleaned, ready to take these vintage floppies in.

However, not all of these disks are off-the-shelf in terms of their formats. Since Jordan did work with a commercial game company, and because there were attempts to prevent wholesale
duplication of these for-sale games at the time, some these floppies have various levels of “copy protection” on them — modifications in how the data is written, in-code checks to analyze the floppy disk’s state and run or not run based on the result, and so on. I’m not here to start a debate on whether this was the right or wrong move at the time — there’s plenty of screen space spent on that discussion elsewhere. But it does translate to a headache for the present day when a straight disk read doesn’t just “work.”

Enter pieces of hardware such as the DiscFerret, CatWeasel, and Kryoflux — all of them modern hardware dedicated to pulling magnetic readings of the floppy disks, eschewing any cares about operating system, structure and copy protection. Think of them as taking a magnetic photograph of the disk. There’s quite a bit of science involved and a lot of debates on what the best approach is for getting the data, but on the whole, the principle is the same: make a floppy drive read the magnetic flux of the floppy, not unlike how a medical scanner approaches the human body, and from that “image,” pull out what the data setup is on the floppy. This resulting magnetic image is huge, size-wise, relative to the original floppies — these 140k (that’s kilobytes) floppies will have a multiple-megabyte magnetic read result from it. But we’re in the space-car future; that mass of data is nothing to us now.

This week, the DiscFerret team has been working overtime, pulling some all-nighters to test and fabricate a hardware setup to do the magnetic readings, and that machinery was packed and FedExed to Jordan yesterday. The in-depth details of what hurdles have to be taken into account with some floppy drive hardware is outside the scope of this already-long post, but rest assured, there are hurdles, and success is not guaranteed.

And let’s make that clear — we have no idea what’s on these floppies! When we bring them in, they could be completely empty (although that is really, really, really unlikely). Factors from quality of manufacture to storage method to phase of the moon could lead to there being lost data. But be assured we’re going in with the respect these artifacts deserve.

See everyone in La-La land!

A clamor immediately arose in the audience, offering to support a Kickstarter campaign for an iOS/Android version of Deathbounce.

I tried to discourage this, pointing out that Doug’s judgment that Deathbounce was unlikely to set the gaming world on fire was well founded even in 1982. Nevertheless, ever since yesterday, people have been coming up to me at PAX and trying to give me money.

It turns out I actually have a disk image of Deathbounce on my MacBook Air.(Derek Moore, childhood friend of voice actor Yuri “Prince of Persia” Lowenthal, did me the favor of copying my 5.25″ Apple II disks a while back. A floppy disk image being only 143K, the shoebox full of disks that took me my entire childhood to amass now occupies barely 2MB on a 350GB hard drive.)

(For those following the Prince of Persia source code saga: Our failure to find the POP source code in this shoebox was what led me to give up hope of ever finding it, until two weeks ago, when my Dad unearthed a forgotten cache of 3.5″ backups in his closet.)

So, rather than do a Deathbounce Kickstarter (I don’t know what scares me more — that it might be a total bust, or that millions of dollars might pour in and then I’d HAVE to make it), I thought it would be simplest to just post the disk image here. This way, anyone who wants to play Deathbounce can.

The cost of the cup of coffee I consumed while typing this post (regular, black) has been funded by the “first Kickstarter dollar” that someone handed me onstage after the talk, plus the five bucks the Penny Arcade security guy offered me to atone for his guilt for playing a pirated disk of Karateka as a kid in England 25 years ago. (Thanks, gentlemen!)

For those who don’t have an Apple II emulator at hand, here’s a short Deathbounce gameplay video, so you can see what you’ve been missing.

(To any programmers out there who’d like to take on the iOS/Android version: I have the source code. You know where to find me.)

See? Way easier than Kickstarter.

My Dad (yep, the same guy who composed the music for the original Karateka and Prince of Persia) called from New York to tell me he was doing some spring cleaning and had shipped me a carton of old games and other stuff of mine he’d found in the back of a closet.

The carton arrived yesterday. My jaw dropped when I saw what was inside.

No, I don’t mean the stacks of Spanish Drosoft versions of POP and Karateka (though those are cool too, especially if you have an Amstrad computer with a cassette player). I mean those three little plastic 3.5″ disk boxes nestled among them… which appear to contain the ORIGINAL APPLE II SOURCE CODE OF PRINCE OF PERSIA that I’ve been searching for, off and on, for the past ten years, pestering everyone from Doug Carlston to Danny Gorlin and everyone who ever worked at Broderbund, and finally gave up hope of ever finding.

I KNEW it wasn’t like me to throw stuff out!

So, for all fifteen of you 6502 assembly-language coders out there who might care… including the hardy soul who ported POP to the Commodore 64 from an Apple II memory dump… I will now begin working with a digital-archeology-minded friend to attempt to figure out how to transfer 3.5″ Apple ProDOS disks onto a MacBook Air and into some kind of 21st-century-readable format. (Yuri Lowenthal, you can guess who I’m talking about.)

This is a crazy busy time (in a good way) with too many projects, so it might take a little while. I’ll document our progress via the twitter and facebook feeds, and I promise, as soon as we can extract something usable, I’ll post it here.

Ilya, Veronika, Jan, Jáchym, Sebastian, Felipe, Robert, Will, Stefano, Chiara, Felix, Alexander, Arnim, Jennifer, Lydia, Lauren, Ravi: You’re absolutely right.

It’s with enormous pleasure that I can finally share this good news: A young French company, DotEmu (who celebrated their fifth anniversary in Paris last night — making them ten years younger than the game) is developing a full iOS version of Last Express, to be released later this year.

Details to follow — but be assured, this will be the complete, original PC game, a deep and immersive real-time interactive narrative offering 20+ hours of game play, with a few additional enhancements to make it more iOS-friendly.

For those who are new to The Last Express, you can read about the original game here. Watch this space, and the official Last Express facebook page, for updates.

My thanks to DotEmu, the original Smoking Car team, and all the Last Express fans who’ve encouraged us to refill the coal tender and stoke the furnace so that this train can leave the station once again, fifteen years later.

I can’t wait!

There was one way, though. You could send a letter to the publisher (the old way, with postage stamps) and hope that it might get to the game creator who might actually read it.

At 17, I didn’t have the chutzpah to think of that — but another enterprising kid named John Romero did. John informed me of this when we finally met, in an elevator at GDC, years after he’d fulfilled his childhood dreams and become one of the best-known game designers on the planet, thanks to Wolfenstein 3D, Doom, and Quake.

John’s first words to me were: “I wrote you a letter. In 1985.”

When I got home, I dug it out of storage. Indeed he had. It was one of the first three or four fan letters I ever got — forwarded by Karateka’s publisher Broderbund Software to my Yale Station post office box, where I was a 20-year-old senior in college. John himself was “17 going on 18,” as he was careful to specify in his letter, perhaps figuring the extra year might cause me to take him more seriously.

John assures me that he has my answer in storage somewhere. I don’t remember what I wrote, but you can read his original letter here. Thanks, John!

I couldn’t resist posting this now, because I’ll be seeing John again next week at GDC. We’ll be on a panel with Tim Sweeney (Epic) and young whippersnappers Adam Saltsman (Canabalt) and Notch Persson (Minecraft), moderated by Jane Pinckard, on the topic of “Back to the Garage: The Return of Indie Development.” Hope to see some of you there!

Anyone who knows me knows I’d rather spend nine hours straight at the computer than fifteen minutes on-camera, but I’m glad Dave and Earl documented this. I think.

Movies are announced early, and often. When a writer sells a pitch, when a director is attached, when a role is cast — all of these stages leading up to making the movie are freely reported and commented on. Even though there’s no guarantee when, or if, an actual movie will ever get made. (See Fathom.) And studios are fine with it.

Video game studios, on the other hand, guard their game development plans like military secrets. It’s not just that they don’t want work-in-progress visuals getting out and giving a less-than-ideal impression of the game. Often, they won’t even confirm that a project EXISTS until it’s almost done, with tens of millions of dollars already spent and the end in sight.

A side effect of this is that, when game developers rub elbows at conferences like GDC, if A should ask B in a moment of drunken camaraderie “What are you working on?” the accepted answer is a big cagey grin and a tease: “Nothing I can talk about!” This is true even if B is the lead designer of Mass Metal Destruction 1 and 2 and remains employed by the same studio. It shouldn’t really surprise anyone that, MMD2 having made half a billion dollars the year before, someone has thought of doing a MMD3. But some things are not to be spoken out loud.

So it’s an exquisite frustration particular to game developers that we spend our time talking (and blogging, and being interviewed) about every aspect of our work EXCEPT what we’re actually working on and are most excited to talk about.

And for me, today, it’s an exquisite joy to finally be able to say this in print:

For the past year, I’ve been working with a small team to develop a new, updated remake of Karateka — the game that began my career 27 years ago.

(If you didn’t happen to encounter Karateka in the early 1980s, you can read its backstory here.)

A New Karateka

Eight years is a long time between games, even for me. Since Prince of Persia: The Sands of Time shipped in 2003, I’ve been busy writing movies, TV, graphic novels, and other non-game projects. It feels great to be hands-on making a game again, and I can’t wait for you to be able to play it.

It’ll be a downloadable game for Xbox 360 and PlayStation 3, out this year. We’re looking at bringing it to other platforms too. I’ll update with more details in the coming months, as we get closer to release.

How different is the new Karateka from the original? It’s closer than the 2003 POP:SOT was to the original, side-scrolling Prince of Persia. But it’s a more radical reinvention than, say, the 2007 XBLA Prince of Persia Classic. The new Karateka is much more than a port; it’s both a remake and a re-imagining of the original game for today’s consoles.

For me as creative director, it’s been an exciting chance to experiment with new gameplay mechanics and ideas that on previous console generations (and on the Apple II) I could only have dreamed about.

Why downloadable and why indie? For a lot of reasons, downloadable just feels right for Karateka. The original was a simple, compact, pick-up-and-play game that didn’t require a lot of tutorial to understand what you had to do. Beating the game was hard, but even little kids could have fun playing it from the first moments. I wanted to honor that simplicity. Jumping from the Atari 400 to a huge triple-A retail console title felt like it would have been too big a leap.

I want to show that a game can be simple fun while also telling a human story in a way that’s emotional, atmospheric, and beautiful. I’ve been encouraged to see gamers embrace downloadable titles like Limbo and Braid — games that stand out because of their design integrity and strong artistic choices, although they were made on modest budgets and don’t represent technological breakthroughs. The industry is changing fast. It’s an exciting time for indie.

And it doesn’t get much more indie than programming a game on a 48K Apple II in my college dorm room, mailing it to a publisher on a 5.25″ floppy disk, and crossing my fingers — which is how Karateka began.

Back to GDC

In a couple of weeks at San Francisco GDC (Game Developers Conference), I’ll be doing a panel with Tim Sweeney (Epic), Adam Saltsman (Canabalt), Notch Persson (Minecraft), and John Romero (Doom) discussing “Back to the Garage: The Return of Indie.” I hope to see some of you there.

After that, it’ll be time to put my head down, get back to work, and get back to not answering questions for the next couple of months.

But now you know at least one of the things I’m working on.

Many thanks to everyone who’s taken the time to comment, or post on twitter or facebook, asking for a new Karateka or sharing your memories of playing the original game. Your encouragement means a lot to me and has helped to get this project off the ground. I truly hope you’ll like the result.

Watch this blog and the Karateka page for updates.

The paper book comes from CreateSpace, a really cool self-publishing service for authors. Basically, we sent them a print-ready PDF and they did the rest. The book weighs in at 323 pages, and looks and feels like a good-quality trade paperback. We’ve priced it at $16.99 (the difference from the ebook versions reflects the printing cost).

You can purchase the book here.

To anyone who’s previously paid for another version of the ebook and would like to have the .epub version for convenience, let us know and we’ll email it to you. Like the PDF, it’s non-DRMed.

Once the dust has settled, I’ll post (and Aaron, Dave and Danica may guest-post) about the results of our grand ebook/self-publishing experiment, and what we’ve learned. Short answer: It was more work than we anticipated — but now that we know how, the next book should be a lot easier. I think.

Also: For readers curious about who some of the people referred to in the journals are, or what became of them, I’ve posted a “who’s who” of players in the making-of-Prince of Persia saga, here.

Many thanks to everyone who’s read the book and reviewed, posted or tweeted about it. The response has been fantastic, and makes it all worth it.

The book isn’t free — we’ve priced it at US$7.99 — but at 300-plus pages, I hope it’s good value. We’re publishing it without any copy protection or DRM, so pirates shouldn’t have much of a challenge. Book sales will help defray the costs of this project and of maintaining the website.

The ebook contains the original Old Journals, plus never-before-published entries leading up to the beginning of The Last Express. You can download a free sample PDF of the first 40 pages, or the full ebook, here.

Thanks to Danica Novgorodoff for designing the book (Danica is the multitalented author of the excellent graphic novel Refresh, Refresh, and designer of many First Second books, including Solomon’s Thieves), and to David Anaxagoras, Ryan Nelson, and Aaron Simonoff for their hard work putting it together. It’s safe to say it turned out to be a lot more work than any of us expected.

How Prince of Persia got made — and almost didn’t

In the ebook, you’ll read what I wrote in my journal on the day I videotaped my kid brother running and jumping to model the prince’s moves; the day I gave up on the project; and the day I decided to finish it after all.

In the seven years from May 1985 to January 1993, Prince of Persia went from a few scribbles on yellow-lined paper to a published, best-selling video game franchise, and I changed from a callow kid into (I thought) a seasoned software entrepreneur. If you’ve read the journals, you know that it was a bumpy ride, and that the game’s eventual success was anything but a foregone conclusion.

Whether you’re a game designer or in another creative field, whether you had an Apple II in the 1980s or weren’t born yet, I hope you’ll find inspiration (or something else of use to you) in this story of how one game got made.

Check out the ebook here.

A request

This ebook is an experiment in many ways. I have no idea how many people will be interested, or how well the non-DRM “honor system” will work. Either way, I’ll post once the dust has settled, and let you know how it went.

If you’ve enjoyed the Old Journals on the site, but don’t feel the urge to own the ebook, you can still support this project by helping us spread the word. Readers like you who take the time to post or tweet about the Old Journals ebook, review it on Amazon, or just tell a friend, will make a big difference in the experiment.

Many thanks!