Development

Ng2-dynamic-dialog 0.0.4 Released

It took me a few tries, but ng2-dynamic-dialog 0.0.4 has now been released and is available to be used by any Angular 2 apps out there.

It is currently built against RC4 (RC5 will be in 0.0.5).

The npm documentation and package is here
https://www.npmjs.com/package/ng2-dynamic-dialog

GitHub project is here
https://github.com/leewinder/ng2-dynamic-dialog

While the links above go into more detail, the following pretty sums up what it can do.

null

Eclipse – Hanging on Android SDK Content Loader or Error Markers

icon-huge

Me and Eclipse (especially on Mac) do not get along. We have a lot of issues, and one of the big ones is the constant hanging when I have to re-open Eclipse to switch projects or branches.

This is probably a post for future me, when I’ve forgotten how to do it and need to Google it again, but maybe it’ll help someone else out.

So, do you open Eclipse and constantly get the IDE hanging on either

  • Android SDK Content Loader
  • Android SDK Resolving error markers

If you do, I’ve found the following resolves it every time (it might also solve a couple of other hangs, but I’ve not been able to test that)

  • Force Quit Eclipse (it probably won’t shut down properly if it’s hung)
  • Open Terminal on your Eclipse workspace
  • Move the .projects folder so we can use it again later
    • mv .metadata/.plugins/org.eclipse.core.resources/.projects projects
  • Reopen Eclipse, your projects should now load
  • Once loaded, close Eclipse again (it’ll probably complain about being unable to save the workspace, that’s fine)
  • Move the projects folder back
    • mv projects .metadata/.plugins/org.eclipse.core.resources/.projects
  • Reopen Eclipse and find you can actually do some work!

Git Workflow Presentation – Intel Buzz Workshop

I travelled to Stockholm this week to deliver a presentation about our Git workflow on Sonic Dash. Originally I was asked to do a talk about the process of converting Sonic Dash to intel based chip sets, but since that would take all of 3 minutes, I worked it into a talk about how we work in general.

The link is possibly tenuous, but at least they had some baring on each other!

Anyway here’s the presentation

Objective-C Swizzling without Side Effects

<tl;dr>

Using method_exchangeImplementations results in a number of unexpected and dangerous side effects which in most cases you’ll want to avoid. You should use method_setImplementation and swizzle a C-style function instead to give yourself more control over your applications behaviour.

</tl;dr>

 

Swizzling is an incredibly powerful, and incredibly dangerous, feature of Objective-C that in the wrong hands can cause a serious amount of problems. A lot of this stems from a lack of understanding about what exactly swizzling is, and how it works. It’s also incredibly common to swizzle using a method that can cause quite serious side effects that if not tidied up can cause quirky and seriously hard to track down bugs.

I initially came across a few of these issues when unit testing our payment pipeline on iOS as I discussed previously. Swizzling worked like a charm, but required a bit of rooting around in the Apple docs to figure out why I was getting the behaviour I was.

Anyway, this post will cover a few of those problems I’ve just described

  1. What exactly is an Objective-C method
  2. Use method_exchangeImplementations at your peril
  3. Prefer method_setImplementation to swizzle without generating any side effects

 

What is an Objective-C method

Objective-C is built on C and as a result it’s methods are actually defined as C-style structs (typedef’d as Method)

struct objc_method
{
     SEL method_name;
     char* method_types;
     IMP method_imp;
}
typedef struct objc_method *Method;

The method_name is the name of the selector (as called by [self selector_name]), the method_types is an encoding of the return value and parameters passed to the function, but the most important entry for now is the method_imp.

This is how it’s defined in objc.h

typedef id (*IMP)(id, SEL, ...);

This means that every Obj-C method is actually just a C-style function that passes through the object, the command and the function parameters. So, if the method called uses any properties or other selectors (calling self. or [self selector_name]), it uses the id object passed through to the function which is where your problems can start to arise.

It’s this structure that’s key to understanding swizzling and the effects it can have.

 

The Consequence of method_exchangeImplementations

The most regular way of swizzling functions is through the use of method_exchangeImplementations. This does exactly as it says, swaps the two method implementations allowing calls to the original method to call the swizzled method instead.

@implementation my_interface

     -(int)swizzle_returns_22
     {
          return 22;
     }

     -(void)display_swizzle_results
     {
          // I’m using SKPayment* as it relates well to the next unit testing post
          SKPayment* payment = [[SKPayment alloc] init];
          NSLog(@“Default call: %d", (int)payment .quantity);
          
          // Get our methods to swap
          Method m1 = class_getInstanceMethod([payment class], @selector(quantity));
          Method m2 = class_getInstanceMethod([self class], @selector(swizzle_returns_22));

          // Exchange them
          method_exchangeImplementations(m1, m2);

          // Call quantity again
          NSLog(@“Replaced call: %d", (int)payment.quantity);
     }
@end

The output of display_swizzle_results is

Default call: 1
Replaced call: 22

But what if display_swizzle_results is modified to include an additional call to the replacement method?

-(void)display_swizzle_results
{
     // I’m using SKPayment* as it relates well to the next unit testing post
     SKPayment* payment = [[SKPayment alloc] init];
     NSLog(@“Default call: %d", (int)payment .quantity);
     
     // Get our methods to swap
     Method m1 = class_getInstanceMethod([payment class], @selector(quantity));
     Method m2 = class_getInstanceMethod([self class], @selector(swizzle_returns_22));
     
     // Exchange them
     method_exchangeImplementations(m1, m2);
     
     // Call quantity again
     NSLog(@“Replaced call: %d", (int)payment.quantity);
     
     // What is the result of this call?
     NSLog(@“Local method: %d", [self swizzle_returns_22]);
}

This new output is

Default call: 1
Replaced call: 22
Local method: 0

0 is probably not what you expected

  • If you didn’t see the 2 lines prior, you’d expect 22
  • If you did see the prior 2 lines, you’d expect 1

So we have two problem here, the first being the fact that you cannot possibly know who’ll use or modify your code in the future, and you’ve altered your code so any call to your functions actually calls something completely different.

The second is the call actually returns garbage.

The first problem is one of documentation and clarification. You could document the heck out of your definition and hope someone in the future bothers to read it, but it’s something a couple of breakpoints and a bit of wasted time would eventually figure out – “oh, it’s not actually calling that function”.

The second would be much harder to track down – “whats it calling and what the hell is 0?”.

So why is the call return 0?

We don’t have the source code to [SKPayment quantity] so we can’t look at the call directly, but that chances are that internally it’s using ‘self’ to access either another selector or another property. Since our implementations are called passing through ‘self’ those properties or selectors that the compiler thinks exists, actually don’t. It’s expecting an object or type SKPayment and instead it’s getting an object of type my_interface.

Who knows if what the internal call to self is calling something that exists in your object or doesn’t, or what the possible side effects will be.

None of this behaviour is ever desired, and should be avoided at all times.

 

Using method_setImplementation Instead

Using method_setImplementation avoids the side effects as it doesn’t swap the functions around (resulting in the call to your client function still calling your client function) and instead it simply replaces your target.

But if you look at the syntax of method_setImplementation it no longer takes a selector (it doesn’t need a new selector so why should it) so switching over isn’t as trivial as just changing the function name.

IMP method_setImplementation(Method method, IMP imp)

An IMP is nothing more than a function pointer that is explicitly passed the two ‘secret’ parameters.

typedef id (*IMP)(id, SEL, ...);

So, in our case, we simply need to define a C-style function to perform the behaviour of our original selector

int swizzle_returns_22_c_style(id self, SEL _cmd)
{
     return 22;
}

Now, to avoid any side effects or unexpected behaviour, here’s the original function again

-(void)display_swizzle_results
{
     // I’m using SKPayment* as it relates well to the next unit testing post
     SKPayment* payment = [[SKPayment alloc] init];
     NSLog(@“Default call: %d", (int)payment .quantity);
     
     // Set our new function
     Method originalMethod = class_getInstanceMethod([payment class], quantity);
     method_setImplementation(originalMethod, (IMP)swizzle_returns_22_c_style);
     
     // Call quantity again
     NSLog(@“Replaced call: %d", (int)payment.quantity);
     
     // What is the result of this call?
     NSLog(@“Local method: %d", [self swizzle_returns_22]);
}

This new output is

Default call: 1
Replaced call: 22
Local method: 22

The behaviour of these two functions is now significantly easier to understand, has zero side effects, and doesn’t result in us overriding behaviour we have no control or understanding of.

So, the next time you need to swizzle any default behaviour you otherwise wouldn’t have any control over, make sure you look at the available API and certainly default to preferring method_setImplementation over method_exchangeImplementations.

Injecting data into Obj-C readonly properties

<tl;dr>

If you want to inject data into an object that only has read-only properties, swizzle the synthesised getter function so you can inject the data you need at the point it’s accessed.

For example

// Interface we want to override
@interface SKPaymentTransaction : NSObject
     @property(nonatomic, readonly) NSData *transactionReceipt;
@end

//
// Returns an invalid receipt
//
NSData* swizzled_transactionReceipt(id self, SEL _cmd)
{
     return @“my receipt”;
}

//
// Test our receipt
//
- (void)test_InvalidTransactionSentViaAPI_VerificationFails
{
     // Create an invalid transaction
     SKPaymentTransaction* invalidTransaction = [[SKPaymentTransaction alloc] init];

     // Replace transactionReceipt with our own
     Method originalMethod = class_getInstanceMethod([invalidTransaction class], @selector(transactionReceipt));
     method_setImplementation(originalMethod, (IMP)swizzled_transactionReceipt);
}

</tl;dr>

 

I recently needed to set up some client side unit tests for our iOS receipt verification server. This server takes a payment object and verifies it with Apple to check if it’s actually a legal receipt and if it is, the content is awarded to the player. Server side, this is pretty simple, but it’s an important server step and should anything happen its possible for people to be locked out from getting their content.

So it’s important we have tests that send legal, invalid and corrupt data to the server and we need to test through the client API otherwise it’s not a test that can be 100% reliable.

Our verify API looks something like the following

+(BOOL) queueVerificationRequest:(NSString*)productId withTransaction:(SKPaymentTransaction*) transaction;

It takes an SKPaymentTransaction because thats the object the client deals with. We’re going to use internal data such as NSStrings or NSData, but it shouldn’t be the clients responsibility to query the SKPaymentTransaction to get the relevant information. Should the structure change, our API also breaks and that’s not acceptable.

So we’re stuck with the SKPaymentTransaction and its API looks something like this

@interface SKPaymentTransaction : NSObject
     @property(nonatomic, readonly) NSError *error;
     @property(nonatomic, readonly) SKPaymentTransaction *originalTransaction;
     @property(nonatomic, readonly) SKPayment *payment;
     @property(nonatomic, readonly) NSArray *downloads;
     @property(nonatomic, readonly) NSDate *transactionDate;
     @property(nonatomic, readonly) NSString *transactionIdentifier;
     @property(nonatomic, readonly) NSData *transactionReceipt;
     @property(nonatomic, readonly) SKPaymentTransactionState transactionState;
@end

Now in our case, we’re interested in using [SKPaymentTransaction transactionIdentifier] and [SKPaymentTransaction transactionReceipt], which both need to be managed inside our verification call so we’ll need SKPaymentTransaction objects that return different values depending on what we’re testing.

And since they’re readonly, we can’t just set them manually.

Initially, I tried to use class extensions, just to add the behaviour I was looking for, as you’ll often add internal extensions to present readonly properties externally but support readwrite properties internally.

@interface SKPaymentTransaction()
     @property(nonatomic, readwrite) NSString *transactionIdentifier;
     @property(nonatomic, readwrite) NSData *transactionReceipt;
@end

Compiles fine but at runtime it generates an unknown selector error. This is because the read only property has already be synthesised, and while the compiler now thinks it can see a setter, at runtime it’s not present.

So, my second attempt was to derive from SKPaymentTransaction and add the functionality there, passing through the base SKPaymentTransaction but using the derived type to set the value.

@interface SKPaymentTransactionDerived : SKPaymentTransaction
     @property(nonatomic, readwrite) NSString *transactionIdentifier;
     @property(nonatomic, readwrite) NSData *transactionReceipt;
@end

Fortunately, the compilers a bit smarter this time and warns me before I even start

error: auto property synthesis will not synthesize property 'transactionIdentifier' because it is 'readwrite' but it will be synthesized 'readonly' via another property [-Werror,-Wobjc-property-synthesis]

At this point I was a bit stuck until @pmjordan suggested I swizzled the transaction identifier and receipts to return the values I’m interested in testing, rather than setting the values directly.

But what needs to be swizzled when we’re attempting to override an Obj-C property defined as follows

@interface SKPaymentTransaction : NSObject
     @property(nonatomic, readonly) NSData *transactionReceipt;
@end

Properties are automatically synthesised (unless explicitly defined) so we’re actually looking at the following selectors

@interface SKPaymentTransaction : NSObject
     // Selector used to return the data
     -(NSData*)  transactionReceipt;

     // This would be defined if we’d specified the property as readwrite
     // -(void)     setTransactionReceipt:(NSData*)receipt;
@end

So at this point, our tests look like the following

//
// Returns an unique invalid receipt
//
NSData* replaced_getTransactionReceipt_Invalid(id self, SEL _cmd)
{
     static int runningId = 0;
     ++runningId;

     NSString* receiptString = [NSString stringWithFormat:@"replaced_getTransactionReceipt_Invalid %d", runningId];
     return [receiptString dataUsingEncoding:NSUTF8StringEncoding];
}

// Checks we fail through the API
- (void)test_InvalidTransactionSentViaAPI_VerificationFails
{
     //
     // Set up ...
     // 

     // Create an invalid transaction
     SKPaymentTransaction* invalidTransaction = [[SKPaymentTransaction alloc] init];
     
     // Inject our invalid receipt method instead of using the default [SKPaymentTransaction transactionReceipt]
     Method originalMethod = class_getInstanceMethod([invalidTransaction class], @selector(transactionReceipt));
     method_setImplementation(originalMethod, (IMP)replaced_getTransactionReceipt_Invalid);

     // Test our verification
     [HLSReceipts queueVerificationRequest:@"test_InvalidTransactionSentViaAPI_VerificationFails" withTransaction:invalidTransaction];

     //
     // Test validation ...
     //
}

As a result of this call, when the library calls [SKPaymentTransaction transactionReceipt] it will instead call replaced_getTransactionReceipt_Invalid which gives us the ability to pass through any receipt we want, including real, invalid and corrupt ones as our tests dictate.

It’s worth noting here that I’m using method_setImplementation rather than the usual method_exchangeImplementations which I’ll explain in a following post soon.

Accu 2014 Conference Notes

I had the chance to go to ACCU 2014 the other week (full conference schedule is here) and I have to say it was one of the best conferences I’ve had the pleasure to attend. And while it did confirm my idea that C++ is getting the point of saturation and ridiculous excess (C++11 was needed, as a result so was C++14, but C++17… Just use a more suitable language if these are the features you need), the talks I went to, on the whole, we’re great.

So I thought I may as well post up the notes I made from each talk – and while they might be a bit of a brain dump of the time, if there’s something that sparks your interest, I’m sure the full presentations will be posted up at some point soon.

 

Get Archaeology
Charles Bailey, Bloomberg LP

Interesting talk looking at some of the more esoteric ways of presenting and searching for information within an existing Git repository. Unfortunately, time was short so the speaker had to rush through the last part, which was, for me, the most relevant and interesting part.

Talk notes

 

Designing C++ Headers Best Practices
Alan Griffiths

For most experienced C++ developers the content of this talk is probably quite basic, as you’d have a large portion of this covered already through sheer trial and error over the years. But clarifying some good practices and interesting side effects was enjoyable.

Talk notes

 

Version Control – Patterns and Practices
Chris Oldwood, chrisoldwood.blogspot.com

Highlevel overview of the various patterns we use when using version control (focused mostly on DVC), some useful examples and some interesting discussion about trust issues…

Talk notes

 

Performance Choices
Dietmar Kuhl, Bloomberg LP

Proof that if you want to know about optimisation and performance, stick to asking people in the games industry.

I’m not posting these notes.

 

Crafting More Effective Technical Presentation
Dirk Haun, http://www.themobilepresenter.com

Really good presentation on how to craft good presentations – some interesting discussions on the make up of the human brain, why certain techniques work and why the vast majority of technical talks (or just talks in general to be honest) do what they do.

Talk notes

 

The Evolution of Good Code
Arjan van Leeuwen, Opera Software

Great talk, not telling us what good code is, but examining a few in-vougue books over the last decade to see where they sit on various contentious topics. Note that when the notes say “no-one argued for/against” it’s just referencing the books being discussed!

Talk notes

 

Software Quality Dashboard for Agile Teams
Alexander Bogush

Brilliant talk about the various metrics Alexander uses to measure the quality of their code base. If you’re sick of agile, don’t let the title put you off, this is valuable reading regardless of your development methodology.

Talk notes

 

Automated Test Hell (or There and Back Again)
Wojciech Seiga, JIRA product manager

Another great talk, this time discussing how the JIRA team took a (very) legacy project with a (very) large and fragile test structure into something much more suitable to quick iteration and testing. When someone says some of their tests took 8 days to complete, you have to wonder how they didn’t just throw it all away!

Talk notes

 

Why Agile Doesn’t Scale – and what you can do about it
Dan North, http://dannorth.net

Interesting talk, arguing that Agile is simply not designed to, nor was it ever imaged to, be a scalable development methodology (where scale is defined as bigger teams and wider problems). Excellently covered why and how agile adoption fails and how this can be avoided to the point where agile principles can be used on much larger and less flexible teams.

Talk notes

 

Biggest Mistakes in C++11
Nicolai M Josuttis, IT-communications.com

Entertaining talk where Nicolai, a member of the C++ Standard Committee library working group, covers various features of C++11 that simply didn’t work or shouldn’t have been included in the standard. When it gets to the point that Standard Committee members in the audience are arguing about how something should or does work, you know they’ve taken it to far.

Talk notes

 

Everything You Ever Wanted to Know About Move Semantics (and then some…)
Howard Hinnant, Ripple Labs

Detailed talk on move semantics which are never as complicated as they (and even the speaker was) are made out to seem. There’s some dumb issues, which seem to be increasing as the C++ standards committee don’t seem to understand the scale of the changes being made, but never-the-less it was a good overview of what is a key improvement to C++.

Talk notes

Git Off My Lawn – Large and Unmergable Assets

I posted up the Git talk myself and Andrew Fray did at Develop 2013 and mentioned I’d have a few follow up posts going into more detail where I thought it was probably needed (since you often can’t get much from a slide deck and no-one recorded the talk).

One of the most asked questions was how we handled large and (usually) unmergable files (mostly in regards to art assets but it could be other things like Excel spreadsheets for localisation etc.). This was hinted to on slides 35 – 37 though such a large topic needs more than 3 slides to do it justice!

To start talking about this, it’s worth raising one of Git’s (or indeed any DVCS’s) major drawbacks and that’s how it stores assets that cannot be merged. Instead of storing history as a collection of deltas (as it does with mergable files) Git simply stores every version as a whole file, which if you have a 100 versions of a 100MB file, it means your repository could be 10GB in size just for that file alone (it’s not that clear-cut, but it explains the issue clearly enough).

While this is a drawback of DVCS in general it’s not necessarily a bad thing.

It’s how all SCM systems handle files that can’t be merged (some SCMS’s do the same with mergable files too – imagine how large their repositories are) but the problem comes with Git’s requirement that a clone pulls down the whole repository and it’s history rather than just the most recent version of all files. Suddenly you have massive repositories on everyones local drive and pulling that across any connection can be a killer.

As an example, the following image shows how a single server/client relationship might work, where each client pulls down the most recent file, while the history of that file is stored on the server alone.

But in a DVCS, the whole repository is cloned on all clients, resulting in a lot of data being transferred and stored on every clone and pull.

Looking at Sonic Dash, we have some pretty large files (though no-where near as large as they could be) most of them PSDs though we have smaller files like Excel spreadsheets that we use for localisation. Since none of these files are mergable and most of these files are large, we couldn’t store them in Git without drastically altering our workflow. So we needed a process that allowed them to be part of the general development flow but without bringing with them all the problems mentioned above.

Looking at the tools available, and looking at what tools were in use at the moment, it made sense to use Perforce as an intermediary. This would allow us to version our larger files without destroying the size of our Git repository but it did bring up some interesting workflow questions

  • How do we use the files in Perforce without over-complicating our one-button build process?
  • With Git, we could have dozens of active branches, how do they map to single versioned assets in Perforce?
  • How do we deal with conflicts if multiple Git repositories need different versions of the Perforce files?

 

By solving the the first point we start to solve the following points. We made it a rule that only the Git repository is required to build the game i.e if you want to get latest, develop and build you only need to use the Git repository, P4 is never a requirement. As a result of this the majority of the team never even opened P4V during development.

This means the P4 repository is designed to only hold source assets, and we have a structured or automated process that converts assets from the P4 repository into the relevant asset in the Git repository.

As an example, we store our localisation files as binary Excel files as thats whats expected by our localisation team but as that’s not a mergable format so we store it in P4. We could write (or probably buy) an Excel parser for Unity but again that wouldn’t help since we’d constantly run into merge conflicts when combining branches. So, we have a one-click script (written in Ruby if you’re interested) that converts the Excel sheet into per-platform, per-language XML files that are stored in the Git repository.

These files are much more Git friendly since the exported XML is deterministic and easily merged. Any complex conflicts and the conflict resolver can resolve how they see fit and just re-export the Excel file to get the latest version.

It also means that should a branch need a modified version of the converted asset they can either do it within the Git repository or roll back to a version they want in P4 and export the file again. The version in P4 is always classed as the master version, so any conflicts when combining branches can be resolved by exporting the file from P4 again to make sure you’re up to date.

Along with this we do have some additional branch requirements that help assets that might not be in Perforce (such as generating Texture Atlases from source textures) but that’s another topic I won’t go into yet.