multipart-message nevyn bengtsson's blog

featured articles 🦄, about, archive, tags

Life in 2019.

hello again. I accidentally let my blog’s domain expire, then my closet server died, then my google cloud copy of it had an expired card and got deleted, while life happened around me. then I sat next to tobias ahlin a few years ago and he convinced me to switch to jekyll. (he reaaalllly loves jekyll.) I started migrating from tumblr, left it without even pushing to github… unhappy about formatting, tried again a year later… and so on.

a year later, the wonderful ted valentin took the whole office with him into the archipelago (including us on parental leave!!) this week, and I got to sit next to tobias again so he could coach me into finish this thing :)

so here we are at nevyn.me. yay, I got a blog again! I’m not really the same person I was in 2015/2016, so I wanted to reboot the blog with a summary of my recent journey. maybe just for a bit of catharsis.

2015: holy shit how did I survive that.

yeah, that was a year alright.

me and my girlfriend of 9 years broke up. a few months later, my dad passed away. while the breakup was pretty mutual and it was nice to change things up, combining the two made me sort of lose myself.

after a long, intense summer, I met my to-be wife Emma at a music festival. the connection was amazing, and we sorta never let go of each other after that.

then my mom passed away too.

then lookback decided to move to silicon valley.

then me and emma got married so we could move there together and continue being ridiculously in love.

that’s…. just too much stuff to go into a single year. i recommend portioning out the good and the bad across a longer period of time, if you can.

lookback goes usa

two and a half years in san francisco, living the startup life. the world grew, I gained a bunch of amazing friends, and I worked myself apart.

san francisco is an amazing city, with such vibrant communities all being destroyed by a selfish, greedy tech industry. biking past four homeless tent camps, and some of the riches companies in the world on my way to work… made me very sad and bitter, broke me a bit.

combine the losses of 2015 while being half a world away from my family, with overworking myself under a not-so-healthy classic startup tech culture, with the broken city around me… it didn’t feel great.

regardless, I made some really great friends. I miss you brian, bruno, kerstin, kristy, bogo, tarjei, aurora, chris, tiff, phil, carolyn, julia, caro, nehal, alex, lea…

family

so emma got a job back in stockholm, so we moved back to sweden. phew. for the two and a half years since the move, it’s been all about different variants of family.

being near my siblings again and meeting them regularly is worth everything to me. it’s an amazing solace. my youngest brother bengan is my closest friend, and being with him is the best.

we reworked our company culture under the amazing henrik, focused on health and human values, slowed down a lot and… started becoming really successful, perhaps much thanks to these new values?? that made me really happy. using the term “family” for work is a common silicon valley dark pattern, but let’s call it… family-like?

I started working from ted valentin’s coworking space knackeriet, and it’s the friendliest place on earth. every single person there is now a dear friend. it’s like a . .. . second family. really, you’re all lovely!!

and in 2018/2019, the actual family of me and emma become a family of three, as our son Theodor joined us ❤❤ they’re my world, especially now as I’m on paternity leave. from july this year until march of 2020, being a dad is my day job.

it’s a lot of work, in a very different way from startup life; but it’s a really welcome change of pace, and being there with my son as he grows up is such an amazing journey ❤❤

hacks, bits and bobs

while parental leave is really not very easy to combine with coding, I’m ridiculously hyped about VR nowadays, so I try to sneak some hacking in. me, tobias kask and admir husovic organize vr the people, a social evening hack hangout group in stockholm for learning VR and working on side projects. it’s a lot of fun :)

and while I do build various retro gaming inspired toys for my son, the project I try to focus on in the few hours a month I actually have time and space to code is alloverse, an overly ambitious VR app platform thatäs been brewing in the back of my head for a long time. I don’t get very far on my own, but some friends are tagging a long, and it’s open source, so it’s getting somewhere :)


so… that’s my life in 2019! let me know on twitter if you enjoyed reading about it, or there’s something in particular you’d want me to blog about:)

Introducing CoreDragon: cross-application drag’n’drop for iPad

I am very passionate about user experience design with direct manipulation. With the advent of multitouch, we have freed ourselves of the computer mouse, and the disconnected interaction of hand movement from cursor movement. Being able to directly touch what we interact with is a great victory, and the major reason why iOS is such an intuitive platform. However, in the design of the iPad, Apple has retained many of the design decisions that were made for a tiny mobile phone, where one can only do one thing at a time. This leads to a lot of modal interactions and sequences of single finger taps, such as the activity view controller, where the entire application is blocked by a modal interaction that you complete by tapping through tedious steps.

The eighties solved this with another piece of direct manipulation: drag and drop. Today, I bring the eighties right back into 2016 with my new open source library CoreDragon. CoreDragon lets you get rid of context menus, modal interactions and even copy paste, by allowing you to mark some areas of your application as things that can be dragged; and other areas as places where you can drop things.

See the video above for a full demo with two sample apps.

nerdsandgamersftw a working bmo super nintendo










nerdsandgamersftw:

A Working BMO Super Nintendo Emulator

3D print designer Mike Barretta built this amazing BMO emulator using an assortment of 3D printed pieces and Super Nintendo buttons, as well as a mixture of electronic equipment. To see this emulator in action, check out the play test of Donkey Kong Country.

You can learn to build your own BMO Emulator via Matt’s Thingiverse. Instructions, needed parts, 3D print files and everything else you need to know can be found on his page. Finally you can have your very own BMO!

The beauty of NSError

Brent Simmons in Init Question:

Let’s say you have an object where initWithSomething could fail due to bad inputs or other error.

Let’s also say that, if it fails, an error should probably be presented to the user.

Brent lists a number of different options. In Programming with Objective-C, there is a golden rule to error handling in ObjC:

  1. If it is a programmer error, use exceptions/assertions.
  2. If it is any other kind of error, use NSError.

This is yet another one of those genius API design decisions that I just love with ObjC. It’s consistent, sane and available everywhere. It leaves very little room for bikeshedding, while still being very convenient to work with most circumstances.

This also eliminates option 4, as it uses assertions for a user/environment error, which would leave your app unusable without any obvious way of fixing it if anything is even slightly wrong. Danger! Don’t do that!

Of the remaining alternatives, I would definitely opt for #1: a failable constructor with in-out NSError parameter. There’s tons of precedent all over Foundation/AppKit/UIKit for this pattern, e g:

  • -[NSData initWithContentsOfURL:options:error:]
  • -[NSString initWithContentsOfURL:encoding:error:]
  • -[NSXMLDocument initWithXMLString:options:error:]

Displaying errors

A very “convenient” option is to just display errors where they happen, and not propagate them. This breaks MVC, and means that your model will likely be too coupled to how the data is displayed, and makes it hard to test and reuse. Make sure to propagate errors up to your controller layer before they’re displayed; either through direct calls and in-out parameters, or delegate protocols. I often have protocols with almost only an “I failed with this error” method.

So, if you’re displaying errors so far from their source, how do you handle them and gracefully get back to a working state, if that working state depends on user interaction? NSError actually solved this problem rather beautifully back on Mac OS, but the whole infrastructure for this wasn’t ported down to iOS: only some bits of pieces followed along. Use these!

You have probably used NSLocalizedDescriptionKey when you’ve constructed your own NSErrors, which is like a ‘subject line’ for the error. There are many more keys you can use to customize errors! There is also NSLocalizedFailureReasonErrorKey, which is like the 'message body’ of the error. NSLocalizedRecoverySuggestionErrorKey lets you tell your user how they could resolve this invonvenient situation. NSUnderlyingErrorKey is very interesting for other programmers: telling them what caused this error in the first place, if that’s available.

But it gets better! The key NSLocalizedRecoveryOptionsErrorKey lets you provide an NSArray of named options for how to solve the error. These are like the button titles of a dialog. You couple it with NSRecoveryAttempterErrorKey, which is any object that conforms to the informal protocol NSErrorRecoveryAttempting.

Hey, this sounds like you could write some generic code that just displays NSError to a user? With title, subject, buttons and actions. Yeah! This is actually what us old Mac-heads had with -[NSApplication presentError::::]. No such equivalent exists in UIKit, but it is easy to build on your own.

LookbackErrorPresenter is an error presenter for iOS, like the one on built into AppKit on Macs, which I just open sourced for your pleasure. You just give it a well-formed NSError with any or all of the above options, and it will be presented and logged appropriately. Here’s some sample usage, straight from the Lookback sources:

NSURL *url = [_group remoteGroupURL] ?: [NSURL URLWithString:_group.endpointURLString];
LookbackRecoveryAttempter *recovery = [[LookbackRecoveryAttempter alloc] initWithNamesAndCallbacks:
	@"Manage experience", ^BOOL{
		GFOpenURL(url);
		return YES;
	},
	@"Dismiss", ^BOOL{ return NO; },
	nil, nil
];
[[LookbackErrorPresenter presenter] presentError:[NSError errorWithDomain:@"io.lookback" code:-18300 userInfo:@{
	NSLocalizedDescriptionKey: @"Experience already published",
	NSLocalizedFailureReasonErrorKey: @"You can delete or manage this experience via the Lookback website.",
	NSRecoveryAttempterErrorKey: recovery,
	NSLocalizedRecoveryOptionsErrorKey: recovery.recoveryOptions,
}] completion:nil];

My approach in controller level code is to create user-friendly NSErrors, populated with ALL the keys above and a recovery attempter, and just display it straight to the user and let them choose how to resolve the error. I can then also include the underlying error, to let more technical-savvy users understand the technical cause, and as a help to future programmers.

Error handling is very important, and if you have the right toolbox, not particularly hard! Just be vigilant, never ignore an error parameter, and propagate errors to where it can be correctly handled.

Additional resources

Writing your own jailbreak tweak

This article is my raw presentation script from CocoaHeads Stockholm 2014-02. It’s written as if I was talking to an audience, not as a blog article, so it might be a bit weird to read.

Hi! I’m nevyn from Lookback. Whenever Marthin sends panicked emails looking for presenters for next Cocoaheads, I look back through the things I’ve encountered recently looking for something interesting. In other words, I am far from an expert on the upcoming subject, but I find it absolutely fascinating. And the topic is: Jailbreak development. I’m going to try to sneak in a product announcement in here somewhere, so don’t zone out!

What does that even mean? Let’s break it down. Jailbreaking has two parts: There’s a jail, and there’s breaking it. The jail is all the security infrastructure of iOS: a virtual room in which your apps run, and very little bad can happen. This infrastructure includes sandboxing, code signing, NX, ASLR, and so on: a seriously impressive layer-cake of security. All of these pieces work together to make sure that only trusted code can be run on your device—only apps signed by Apple or by you as a developer for your own device—and even if that code is compromised, there is a very limited range of possible consequences; for example, even if you happen to have a buffer overrun vulnerability in your app, a hacker can’t use that bug to install malicious code (like a virus) on your phone.

I could go into details of how all this works, but to allow us to move on, instead I deeply recommend Charlie Miller et al’s iOS Hacker’s Handbook. It’s an excellent guide to both basic security, how to write secure iOS applications, the security infrastructure of both FreeBSD, Mac OS X, and iOS, and how all of those have previously been broken.

That gets us on to the “breaking” in “jailbreaking”. What does “breaking” mean in this context, anyway? Jailbreaking means finding operating system bugs that allow you to expose vulnerabilities by writing exploits to disable every single one of the defenses put in place.

For example, the iOS 7 jailbreak “evasi0n” works like this:

  1. The WWDC 2012 app has code signing that allows it to run anywhere forever. However, it’s still just a normal app within a sandbox, so it can’t make changes to the operating system.
  2. A bug in the way the app installer handles symbolic links allows us to run iOS’s file server instead (used to make backups, normally).
  3. A bug in the dynamic linker allows a flag to completely disable the sandbox. However, we’re still running as the user, not as root, so we can only modify the user’s apps and documents, not the operating system.
  4. A bug in the way the file server handles symbolic links gives us raw access to the harddrive.
  5. And at this point, we can write whatever we want to the disk. The sandbox is completely disabled, allowing us to run any software we want.

This should be scary. This means that on a jailbroken device, a mail attachment could contain a virus, and all the gatekeepers would be asleep: the virus could execute, and it could infect your phone and spread it to all your contacts. Just like on a normal computer.

…

and that’s sort of the point. You own your iPhone. It’s a computer, just like any other. You shouldn’t have to pay Apple *additional* money just to be able to install whatever software you want on it.

And I really mean *whatever* software. On a Mac you can also whatever you want, but there never really sprung up a big community of modders. There was CandyBar, Audio Hijack and a few others.

But on iOS, every jailbreak comes with Cydia, and a whole lot of infrastructure written by Saurik and other awesome hackers. Every piece of the system has a category of modifications on his store. Saurik likes calling “remixing software”, which I think is a great term.

How would one get started with jailbreak development, with remixing software?

Large parts of the jailbreak community is based on the idea that you shouldn’t need a Mac to do development, so they’re based on Makefiles and other traditional Unix development tools. Me, I prefer my GUIs. If you’re like me, you’ll appreciate iOS Open Dev, which is an Xcode plugin package with awesome templates to help you get started. One might think it’s just a bunch of unmaintained scripts (as is often the case with obscure dev communities), but I’ve found it to be very well put together.

After installation, iosopendev might not work right off the bat. If it doesn’t, you might to tell it to use the latest iOS SDK:

iod-setup sdk -d /Applications/Xcode.app/Contents/Developer -sdk iphoneos

Restarting Xcode and making a new project now gives a ton of new options. They’re all based on the idea that you want to put code into other people’s apps. Some interesting ones are:

  • Action Menu Plugin. Always wanted to be able to select text and have an option to make it sparkle? Well, why don’t you build it!
  • Tweak. A tweak is a plugin that is loaded into apps that aren’t supposed to have plugins. You can make a tweak that loads into every single app, or just into Springboard, which is the app that launches apps and does all the other iOS-y stuff. This is the most interesting one, so I’ll get into more detail on it later.
  • Activator Listener. A crazy infrastructure which lets you bind any action to any handler. For example, if you make, I dunno, a tweak that lets you record the screen of any app, you might want to add an Activator to it so that the end user can choose which action should start the recording. And it could be ANY action. This is just a small sample. When locking the phone. When triple-tapping the lock button. When inverse five-finger swiping the screen. When dragging in with two fingers from top-right of the screen. You get the point.
  • Library. Hey look, you can actually make dynamic libraries on iOS!
  • Tool. And command line tools! It’s as if this is some kind of UNIX operating system!
  • You can make your own Notification Center widget, just like Weather and Stocks.
  • Toggles. Before Control Center in iOS 7, jailbreakers enjoyed the same features (and many more) using SBSettings toggles. You can make your own.
  • XPC Service! Now this is even starting to look like Mac OS X! How odd.

Ok, so before we jump into live coding, I’d like a volunteer app. If you have an app on the AppStore and want to be horrified, yell the app name out loud!

While that’s downloading, let’s make a tweak. Primarily, a tweak is a uniform way of injecting code into apps using a library called Cydia Substrate. It will let you specify which apps you want to inject in, for example by checking for the existence of a class. This filter here will inject into any app that has the hypothetical class SpotifyViewController in it. You can also say “only inject into SpringBoard”, or “into any UIKit app”, or something else.

Once inside the target app, what can you do?

Well, a tweak is also method swizzling made into an art form. To make it really, really easy to swizzle stuff. two optional frameworks have emerged.

Logos is the oldest one, and it is a bit crazy. It’s a custom preprocessor on top of Objective-C with the extension “.x”. You tell it which class you want to swizzle, implement your replacement, and then you can optionally call the original implementation with %orig. It has support for many other things, but I think I’ll leave it at that.

The alternative is CaptainHook, which uses C++ template classes and macros to accomplish the same thing. It’s not less crazy. Here we are logging every call to [NSString writeToFile:], and then calling the original implementation.

So let’s write a very simple tweak. It’ll create a class called “ListClasses” when the app starts, using +load. We’ll wait for the app to finish launching properly before we do anything. At that point, we’ll add a button to the window after a second, to let the UI settle. It’s going to present a custom view controller on top of the app.

This’ll be a table view controller. We will list ALL classes available to the app, and get their names. Then we’ll just list those in a table view. Build and run, aaand… Let’s have a look inside the audience app.

So, these are real class names, and I could instantiate any one of them if I wanted, and call any method I wanted on them.

Class names aren’t quite enough to know what to swizzle though: we want method names, and preferrably, even headers. Luckilly, or unfortunately if you want to keep secrets, Objective-C is very verbose in what kind of information is available at runtime, since it’s such a dynamic language. There’s a tool called class-dump that will take an app or a library, and output headers, much like this. However, all app-store apps are encrypted and aren’t easily class-dumped. First, they must be cracked using a tool using “Clutch”. This is also the same way that apps are pirated, so please don’t do that :(

Let’s do a classdump of the audience app. I will SSH into my phone, as if it was a regular UNIX computer o_O

This is a good time to mention that development on device is going to be very painful if you don’t enable SSH. SSH lets you get a command line shell right into your phone. However, you should change the password immediately after installation, since all iPhones have the same default password: alpine. You’ll also want to add a ssh key to your phone, so that iosopendev can install tweaks onto your phone directly from Xcode. If you add your phone’s IP address to your .bash_profile file, you can press command-shift-I to install tweaks.

You will probably also want the tool deviceconsole, which is like Xcode organizer’s console log, except as a terminal app which means it’s searchable, fast, and just generally awesome in every way that Xcode isn’t.

There’s so much more to talk about. How to publish your creation, how to debug, how to write activators, and much more. However, my time is up so talk to me afterwards if you want to know more.

I’d like to finish off with the tool I built that made me learn all this. You probably saw my previous presentation on Lookback, so now I’d like to show you Lookback Anywhere. Lookback Anywhere is very simple: I’ve taken the Lookback SDK, linked it into a tweak, and added two activator actions. That means Lookback is loaded into every app on your phone, and you can record them without having to install the SDK. The same concept could be applied to any developer tool: Reveal, PonyDebugger, Spark Inspector and others could be installed into any app very easily.

So I’d like to show you here how I can record in this audience app… And voila, it’s uploaded to lookback.io.

Lookback Anywhere is not ready for release, but you can install a beta on your phone by adding a custom source in Cydia.

mastervone welcome program you have only two


mastervone:

Welcome, program! You have only two choices: Games, or immediate deresolution! Luckilly, you can bring one or two, or even three of your “friends" onto the Grid. Not so lucky: only one of you will survive.

Gridlike is now available for iPad on the Apple App Store. It’s a two-to-four player lightbike game that we’ve been working on on and off since before Tron: Legacy was released.

We released a game! :D We were basically done with this game after a week-long game hacking lan party three years ago, but weren’t quite happy with the result, and there was this one annoying collision bug we never got around to fixing. Every game making lan party since, we’ve said that we’ll patch it up and release it, but things like CLOP (don’t click.), Candy Box (don’t click!!), Deathtroid (do click!) and just general procrastination got in the way. (We’re horrible procrastinators).

Last week, we had another game making LAN, (chronicled in my dev diary), and managed to both finish up Gridlike and start a new game, Tanks!

We’ve set the price to free for the first week, so go check it out! Let us know if anything’s weird, buggy or boring.

Continuously playing music in the background on iOS

tl;dr: Have an active background task and audio session running continously whenever the user is semantically playing music, so code can execute while empty buffers are being refilled.

For the longest time since multitasking was introduced in iOS 4, Spotify occasionally had problems with music playback stopping against the user’s wishes. See, iOS applications can only execute in the background when at least one of six conditions are true:

  • App has background mode “audio” in Info.plist and is playing audio, with a suitable audio category active
  • Equivalent, but for getting location data
  • Equivalent, but for talking to external accessories
  • App is performing magic VoIP voodoo
  • App is a Newsstand app and is downloading content
  • App explicitly started a ten minute limited “background task”

If you’re a streaming music service, music to be played in the background won’t always be immediately available. Say your music buffer runs out, your device is on EDGE, and it takes a few seconds to get the music data: silence. Or even if you’re not streaming music, perhaps you need to tear down and set up your audio subsystem in between tracks, e g because they’re different kinds of tracks. What should a well-behaved iOS app do in this situation?

When I realized what the problem is, my first approach was to detect situations when music would not be playing, try to pre-empt them, and start a background task just before music stops playing and then end it when music is back again. If this sounds fragile and stupid to you, you’re absolutely right. I’ve heard other developers come to the same solution, but don’t do it: you’ll miss edge cases, it’ll be buggy, and it’s so overly complex.

The epiphany came when I realized what the audio background mode actually means for background tasks: background tasks become no-ops, with infinite lease time, as long as audio is playing. Only when there is no audio playing does the timer start counting, from ten minutes. Start playing again and the lease is back at infinite time.

The solution is then obvious: whether your app is playing audio is uninteresting. The only interesting information is whether the user currently wants music to play. If she does, start an audio session and a background task, and don’t stop them until the user wants music to stop. In between, it’s up to other subsystems to make sure that audio is playing. With the continous background task, we allow these subsystems to continue to work, even if they can’t play music this very moment. As long as they don’t take playback pauses that are longer than ten minutes, we’re all good.

This is a very simple approach, and should work no matter how your underlying audio playback code works. Below, you can find some sample code implementing my approach.

(If you’re thinking you haven’t seen this problem in Spotify in a long while, you’re right: I wrote this code two years ago and have been thinking I should blog it since then. My apologies for not doing so earlier.)

Thinking alike: await/async in Python 3

Thinking alike: await/async in Python 3

Guido van Rossum’s 2013 PyCon keynote presents Python 3.4’s major new feature: coroutines as a language feature for working with asynchronous operations. I wonder where I’ve seen that before? :)

Async/await is such an amazingly simple way to express asynchrony, and I’m very happy to see more languages adopt the thinking. Also, python is *finally finally finally* standardizing on a runloop API. That has been a crazy big gap in the standard libraries and frankly I don’t understand how people have been able to write Python apps without it…

Example python code with the new API below. “yield from” is identical to “await” in C#.

def getresp():
    s = socket()
    yield from loop.sock_connect(s, host, port)
    yield from loop.sock_sendall(s, b'xyzzy')
    data = yield from loop.sock_recv(s, 100)