Over doing it with methods in C++

A fun fact about the ‘+’ operator is that it can be applied a lot of different types. We can use it on integers, on doubles, even on strings. For each of these, it does some kind of addition. This is method overloading. There are multiple different version of ‘+’ with different signatures, so we can have an abstract notion of addition that works across different types. The great thing is, we can do this in our own code!

What is Overloading?

Overloading a method is when you have more than one method with the same name but different argument types in a class. Something like

class Greeter {
public:
    std::string say_hello() {
        return "Hello";
    }

    std::string say_hello(std::string name) {
        return "Hello " + name;
    }
}

Given a Greeter object, a call to say_hello with no arguments will call the first method and a call to say_hello with a string argument will call the second method.

So we can make as many methods name say_hello as we want in this class with different parameters. However, we cannot have an overload that differs only in return type, like this

class Greeter {
public:
    std::string say_hello() {
        return "Hello";
    }

    // Compiler error!
    bool say_hello() {
        return true;
    }
}

With this code, if you had a Greeter object, and you call say_hello the compiler wouldn’t know which method you are referring to. This is because a call to the say_hello method, might be part of a larger expression, with more method calls, like so:

f(g(greeter.say_hello()) + 5) * g(a)

when this happens, the compiler works recursively upwards, from the deepest method call to the topmost, resolving the types as it goes. So the compiler uses the return type of a method to resolve the type of the parameters of the method at the next level up. So it always needs to know what the return types are.

Don’t hide your wonderful code!

Suppose now we have a base class and a subclass, like this

class BasicGreeter {
public: 
    std::string say_hello() {
        return "Hello";
    }
}

class PoliteGreeter : BasicGreeter {

}

If you have a PoliteGreeter object, you can call the say_hello method from it’s base class. But, suppose we added a say_hello method to PoliteGreeter itself, like so

class BasicGreeter {
public: 
    std::string say_hello() {
        return "Hello";
    }
}

class PoliteGreeter : BasicGreeter {
public: 
    std::string say_hello() {
        return "Hello there good sir";
    }
}

This will compile happily. But, because both say_hello methods have the same signature, the say_hello method in PoliteGreeter is hiding the say_hello method in BasicGreeter. If we call say_hello from a PoliteGreeter object we will get the PoliteGreeter version of that method, not the BasicGreeter version. However, we can still access the BasicGreeter version of the say_hello method by using the base class name directly, like so

PoliteGreeter greeter;
greeter.BasicGreeter::say_Hello(); // This is the sub class say_hello method!

What happens if we try to overload a base class method in a subclass?

class BasicGreeter {
public: 
    std::string say_hello() {
        return "Hello";
    }
}

class PoliteGreeter : BasicGreeter {
public: 
    std::string say_hello(std::string name) {
        return "Hello there " + name;
    }
}

Bad news, even though they have different signatures, the base class method is still hidden! This is quite counter-intuitive to me. In fact, the compiler could make both methods visible in the PoliteGreeter class. But, the C++ standard does not allow it, for confusing and obscure reasons, that I do not understand.

The good news is that we can still use the previous trick to access the subclass method.

PoliteGreeter greeter;

greeter.say_Hello(); // Compiler error! The BasicGreeter say_hello method is hidden!
greeter.BasicGreeter::say_Hello(); 

We can also add a using statement to the PoliteGreeter class itself, that will make the say_hello method in BasicGreeter visible from a PoliteGreeter object.

class PoliteGreeter : BasicGreeter {
public: 
using BasicGreeter::say_hello;

    std::string say_hello(std::string name) {
        return "Hello there " + name;
    }
}

If we want to call the say_hello method from BasicGreeter inside PoliteGreeter without making it visible outside of the class we can reference the subclass like this:

class PoliteGreeter : BasicGreeter {
public: 
    std::string say_hello(std::string name) {
        return BasicGreeter::say_hello() + name;
    }
}

Staying Safe in C++

What is Exception Safety?

Exceptions are gross. One of the problems with exceptions is that, as they break out of the normal control flow, they can potentially leave our program in a bad state. How do we avoid this?

Well, we can guarantee that we never throw an exception, either because our code is perfect, or, we catch and smoothly handle all exceptions at the point they are thrown. This is a worthy goal, but might not be entirely practical. So the alternative is to follow the philosophy of exception safety. This is the design philosophy that says that we can throw exceptions, but, when we do, we must leave things in a good state.

There are two different flavours of exception safety, strong exception safety and weak exception safety. Strong exception safety is a guarantee that when an exception is thrown during some operation, the overall state will be back to what it was before the operation started. In other words, our methods are atomic or transactional. Weak exception safety is the guarantee that when we throw an exception we will leave things in a valid state. In particular, both forms of exception safety guarantee that we will not leak any resources.

Usually we manage state via classes, and that is also how we think about exception safety. How we make a class exception safe depends on the state that our class manages, and the functionality it exposes via it’s methods.

An Example of Exception Safety

Here’s an example from a job interview I did recently. Suppose we have two strongly exception safe classes: ResourceManager1 and ResourceManager2. In our code we want to define another class, which we, optimistically, name, SafeCopyingClass, like so:

class SafeCopyingClass
{
private:
   int a;
   ResourceManager1 b;
   ResourceManager2* c;
}

The question is, how can we implement a copy constructor for this class in an exception safe way?

We have to copy three member variables. We know that copying an integer like the member a should not throw. This means that we should leave the copy of a until the very end, that way if something goes wrong with the other two copies, we won’t have to roll it back. So, to start, we are going to have something like this:

SafeCopyingClass(const SafeCopyingClass& other)
{
   /// lots of copying logic that might throw
   a = other.a;
}

Now what about copying the members b and c? Suppose we naively copy the objects b and c as below.

SafeCopyingClass(const& SafeCopyingClass& other)
{
   b = ResourceManager1(other.b);
   c = new ResourceManager2(*other.c);
   a = other.a;
}

If the copying of other.b throws, then we are still in the original state so we can just let the exception escape. However, if that first copy succeeds, and then the copy of the RescourceManager2 object throws, then we are not in the original pre-copy state. Indeed, things are even worse than that, we cannot safely get back to the original state. Anything we do to try and restore the pre-copy state, would involve copying ResourceManager1 objects, which would potentially throw again.

So, instead, we are going to copy the ResourceManager2 object first. We are in luck here, because c is in fact a pointer, which is a primitive type, and, although copying the object it points to could throw, copying the pointer itself should not. This means that, we can split the operation into a copy that may throw but leaves our state unchanged, and an assignment that changes our state but should not throw. Concretely, we first copy the other.c object and store the result in a temporary pointer, and then separately assign value in that temporary point to c.

We do this copy of the underlying ResourceManager2 object first, if this throws, as we have not altered the state of the SafeCopyingClass object, we can let this exception escape. The assignment of the value in other.c to c, alters the state so it should go at the end, after all the code that may throw. So our copy constructor now looks like this:

SafeCopyingClass(const SafeCopyingClass& other)
{
   ResourceManager2* temp_c = new ResourceManager2(*other.c);
   // more copying logic that might throw
   c = temp_c;
   a = other.a;
}

Now we just have to deal with the copying of our ResourceManager1 object b. After creating the temporary pointer for our other.c but before we assign that pointer to c, we are still in the original state, so we can copy b and allow exceptions to escape, at this point, like so:

SafeCopyingClass(const SafeCopyingClass& other)
{
   ResourceManager2* temp_c = new ResourceManager2(*other.c);
   b = ResourceManager1(other.b);
   c = temp_c;
   a = other.a;
}

However, there is still one issue with our code. We have a potential resource leak, and that is not only bad practice, it is not exception safe. Suppose, when we copy the other.b object an exception is thrown. Then control will leave this copy constructor. However we created an object of type ResourceManager2. The pointer temp_c is lost, but the object will live on, so we are leaking resources.

So, if the copy of other.b throws, we need to delete the object pointed to by temp_c. To do this, we wrap the ResourceManager1 copy in a try catch block. Then if an exception is thrown we can clean things up and rethrow the exception. So, here his our finished copy constructor:

SafeCopyingClass(const& SafeCopyingClass other)
{
   ResourceManager2* temp_c = new ResourceManager2(other.c);
   try
   {
      b = ResourceManager1(other.b);
   } 
   catch(...)
   {
      delete temp_c;
      throw; 
   }
   c = temp_c;
   a = other.a;
}

What is the general strategy here? We split our copying into stuff that may throw and stuff that should not throw. We do everything that might throw before everything that will not throw. That makes any potential rollback a lot simpler. When we have a pointers, we should copy the underlying objects first and store the result in temporary pointers. After that, we do the rest of the potentially throwing copies, this is where we have to be aware of potential resource leaks and non trivial rollbacks.

Constructors and Destructors in C++ – Part 3

We’ve talked a lot about how construction and destruction work. Now, finally, it’s time to see how they go wrong. Firstly, let’s say that an object is being constructed and an exception is thrown. In this situation the classes destructor is not called. This, in general, should not be a problem. The only exception throwing code you should have in a constructor is code that manages some resource. And, you should manage at most one resource per class. So if an exception is thrown in your constructor, it means that the resource you are managing has not been setup and so the destructor does not need to be called.

So if we have a class like this:

class Base
{
public:
	Base()
	{
		std::cout << "Base Constructor" << std::endl;
		throw std::exception();
	}

	virtual ~Base()
	{
		std::cout << "Base Destructor" << std::endl;
	}
};

and we run the following main function,

int main()
{
    try
    {
        Base baseObject;
    }
    catch(std::exception& e)
    {
        std::cout << "Exception caught" << std::endl;
    }
    return 0;
}

we will see:

Base Constructor
Exception caught

As you see, no calls to the destructor. Now suppose, with the same main function, we throw an exception from a derived class like so,

class Base
{
public:
	Base()
	{
		std::cout << "Base Constructor" << std::endl;
	}

	virtual ~Base()
	{
		std::cout << "Base Destructor" << std::endl;
	}
};

class Derived : public Base
{
public:
	Derived() : Base()
	{
		std::cout << "Derived Constructor" << std::endl;
		throw std::exception();
	}

	Derived(int i)
	{
		std::cout << "Derived with parameter Constructor" << std::endl;
	}

	virtual ~Derived()
	{
		std::cout << "Derived Destructor" << std::endl;
	}
};

Then we will see:

Base Constructor
Derived Constructor
Base Destructor
Exception caught

The Derived destructor is not called, but the Base Destructor is. This is because by the time the exception is thrown in the Derived class, we have completely the construction of the Base class, so that will need to be destroyed as normal.

Local objects that are constructed within the constructor will be destroyed as normal when an exception is thrown. So, for example code like this:

class Simple
{
public:
	Simple()
	{
		std::cout << "Simple Constructor" << std::endl;
	}

	virtual ~Simple()
	{
		std::cout << "Simple Destructor" << std::endl;
	}
};

class Basic
{
public:
	Basic()
	{
		std::cout << "Basic Constructor" << std::endl;
                Simple simpleObject;
		throw std::exception();
	}

	virtual ~Basic()
	{
		std::cout << "Basic Destructor" << std::endl;
	}
};

when executed via,

int main()
{
	try
	{
		Basic basicObject;
	}
	catch(std::exception& e)
	{
		std::cout << "Exception caught" << std::endl;
	}
	return 0;
}

will produce:

Basic Constructor
Simple Constructor
Simple Destructor
Exception caught

So, as we have seen, you can throw exceptions from the constructor. You cannot however throw an exception from a destructor. Try building the following code,

class Simple
{
public:
	Simple()
	{
		std::cout << "Simple Constructor" << std::endl;
	}

	virtual ~Simple()
	{
		std::cout << "Simple Destructor" << std::endl;
                throw std::exception();
	}
};

You should get a warn telling you that destructors are “noexcept” in C++. If you run this code as before, you will get:

Simple Constructor
Simple Destructor
terminate called after throwing an instance of 'std::exception'
  what():  std::exception
Aborted (core dumped)

C++ will always terminate when an exception like this is thrown in a destructor. The reason is simple. Suppose we throw an exception and then start unwinding the stack. If, when we destroy an object on the stack it’s destructor threw another exception, there is no way to catch both exceptions. We cannot catch one exception and abandon the other, so the program gives up and terminates whenever an exception is thrown in a destructor.

Constructors and Destructors in C++ – Part 2

In our previous post we discussed constructors and destructors for local objects. Now, suppose we have the same hierarchy of classes, Base, Derived and DerivedAgain, and we constructed a DerivedAgain object the heap, like so:

int main()
{
    DerivedAgain* derivedAgainObject = new DerivedAgainObject();
    return 0;
}

When we run this we will see the following,

Base Constructor
Derived Constructor
DerivedAgain Constructor

The constructors were called in the same order as before. However, because our DerivedAgain object was created on the heap the destructor was never called. If we want to destroy our object we must explicitly use delete.

int main()
{
    DerivedAgain* derivedAgainObject = new DerivedAgainObject();
    delete derivedAgainObject;
    return 0;
}

The above will output the following to the command line,

Base Constructor
Derived Constructor 
DerivedAgain Constructor
DerivedAgain Destructor
Derived Destructor
Base Destructor

Let’s consider what happens when we have a pointer of type Derived that points to an object of typed DerivedAgain. Say we run:

Derived* derivedPointer = new DerivedAgain();
delete derivedPointer;
return 0;

When we run this code we will see:

Base Constructor
Derived Constructor
DerivedAgain Constructor
Derived Destructor
Base Destructor

The constructors are the same, because we are creating a DerivedAgain object as before. However, now we are calling delete on a pointer of type Derived. So only the Derived and Base class destructors are called.

We can avoid this problem by making our destructors virtual.

class Base
{
public:
        Base()
        {
                std::cout << "Base Constructor" << std::endl;
        }

        virtual ~Base()
        {
                std::cout << "Base Destructor" << std::endl;
        }
};

class Derived : public Base
{
public:
        Derived() : Base()
        {
                std::cout << "Derived Constructor" << std::endl;
        }

        virtual ~Derived()
        {
                std::cout << "Derived Destructor" << std::endl;
        }
};

class DerivedAgain : public Derived
{
public:
        DerivedAgain() : Derived()
        {
                std::cout << "DerivedAgain Constructor" << std::endl;
        }

        ~DerivedAgain()
        {
                std::cout << "DerivedAgain Destructor" << std::endl;
        }
};

Now when we run,

int main()
{
        Derived* derivedPointer = new DerivedAgain();
        delete derivedPointer;
        return 0;
}

we will see:

Base Constructor
Derived Constructor
DerivedAgain Constructor
DerivedAgain Destructor
Derived Destructor
Base Destructor

Even though we have a pointer of type Derived, because the destructor is virtual, when we call delete, the DerivedAgain destructor gets called. What’s interesting here, is that we always call all the destructors upwards through the type hierarchy, the use of virtual ensures that we start at the right level. For this reason, we should usually make our destructors virtual.

We can even make our destructors pure virtual. However, we will have to give this pure virtual destructor a body. It is a common misconception that a pure virtual function necessarily can’t have a body, this is not true. However pure virtual destructors are probably one of the only examples of when you would want to do it. The only reason we would want to make a destructor pure virtual is that we want to make a class abstract and the destructor is the only method. In my opinion this is a very unlikely to happen, so it’s not something we should really worry about.

In the next post we will have a look at what happens when things go wrong in construction and destruction.

Constructors and Destructors in C++ – Part 1

We’re going to think a little bit about constructing and destroying objects in C++. Let’s say we have three very simple classes, Base, Derived and DerivedAgain.

class Base
{
public:
	Base()
	{
		std::cout << "Base Constructor" << std::endl;
	}

	~Base()
	{
		std::cout << "Base Destructor" << std::endl;
	}
};

class Derived : public Base
{
public:
	Derived()
	{
		std::cout << "Derived Constructor" << std::endl;
	}

	~Derived()
	{
		std::cout << "Derived Destructor" << std::endl;
	}
};

class DerivedAgain : public Derived
{
public:
	DerivedAgain()
	{
		std::cout << "DerivedAgain Constructor" << std::endl;
	}

	~DerivedAgain()
	{
		std::cout << "DerivedAgain Destructor" << std::endl;
	}
};

Now suppose we run the following code,

int main()
{
	DerivedAgain derivedAgainObject;
	return 0;
}

The output will be:

Base Constructor
Derived Constructor 
DerivedAgain Constructor
DerivedAgain Destructor
Derived Destructor
Base Destructor

Why is this? Well, to construct the DerivedAgain object, first we must construct aDerived object and to construct a Derived object, first we must construct a Base object. Similarly, when the object goes out of scope, to destroy it, first the DerivedAgain destructor is called, then the Derived destructor and then the Base Destructor. So, when creating an object we step down through the inheritance hierarchy, and when destroying it we step back up through the hierarchy.

We can call these constructors explicitly. Indeed the above code is exactly quivalent to:

class Base
{
public:
	Base()
	{
		std::cout << "Base Constructor" << std::endl;
	}

	~Base()
	{
		std::cout << "Base Destructor" << std::endl;
	}
};

class Derived : public Base
{
public:
	Derived() : Base()
	{
		std::cout << "Derived Constructor" << std::endl;
	}

	~Derived()
	{
		std::cout << "Derived Destructor" << std::endl;
	}
};

class DerivedAgain : public Derived
{
public:
	DerivedAgain() : Derived()
	{
		std::cout << "DerivedAgain Constructor" << std::endl;
	}

	~DerivedAgain()
	{
		std::cout << "DerivedAgain Destructor" << std::endl;
	}
};

Let’s change things up a little bit, suppose we have the following situation.

class Derived : public Base
{
public:
	Derived()
	{
		std::cout << "Derived Constructor" << std::endl;
	}

	Derived(int i)
	{
		std::cout << "Derived Constructor With Parameter" << std::endl;
	}

	~Derived()
	{
		std::cout << "Derived Destructor" << std::endl;
	}
};

class DerivedAgain : public Derived
{
public:
	DerivedAgain() : Derived(5)
	{
		std::cout << "DerivedAgain Constructor" << std::endl;
	}

	~DerivedAgain()
	{
		std::cout << "DerivedAgain Destructor" << std::endl;
	}
};

Now we are explicitly calling a non-default constructor from the Derived class. All the other implicit calls to default constructors remain the same. When we run the main function as before, we will see:

Base Constructor
Derived Constructor with parameter Constructor
DerivedAgain Constructor
DerivedAgain Destructor
Derived Destructor
Base Destructor

You cannot however do something like this:

class DerivedAgain : public Derived
{
public:
	DerivedAgain() : Base()
	{
		std::cout << "DerivedAgain Constructor" << std::endl;
	}

	~DerivedAgain()
	{
		std::cout << "DerivedAgain Destructor" << std::endl;
	}
};

Here, we are trying to call the Base constructor from the DerivedAgain class. This will not compile, we can only call constructors from the classes that we directly inherit from.

So far we have only looked at constructors and destructors for local objects, that is, without using the new keyword. In part 2, we will look at what happens when we construct and destroy objects dynamically.

What are Dependency Injection Frameworks and Why are they Bad?

What is Dependency Injection?

Let’s suppose we are writing some code. This is part of a big legacy code base and we are using an object oriented language like Java or C#. Say we would like to value some stocks.

Suppose we have a class, StockValuer, that takes as its inputs, various interfaces, say, an IInterestRateProvider, a INewsFeedReader, and a IHistoricPriceReader. These interfaces are implemented by theInterestRateProvider, NewsFeedReader and IHistoricPriceReader classes respectively. Now each of these types in turn will have arguments they depend on, but we will elide that for now. So we will set it all up, with something like:

IInterestRateProvider interestRateProvider = new InterestRateProvder(...);
INewsFeedReader newsFeedReader = new NewsFeedReader(...);
IHistoricPriceProvider historicPriceReader = new HistoricPriceReader(...);

IStockValuer stockValuer = new StockValuer(interestRateProvider, newsFeedReader, historicPriceReader);

So we have one top level class, the StockValuer, and we pass various other objects into it, that provide the functionality it needs. This style of programming is called Inversion of Control or Dependency Injection.

Usually, when we write code like this, we test it by passing fakes or mocks of the various interfaces. This can be really great for mocking out the kind of low level stuff that it is usually hard to test like database access or user input.

These style of programming goes hand in had with the factory pattern. We can see as much above, we have written a factory to create our StockValuer!

What are Dependency Injection Frameworks?

There is another way for us to create our StockValuer. We can use a dependency injection Framework like Spring in Java or Castle Windsor in C#. We will no longer have a factory that explicitly builds up our StockValuer. Instead, we will register the various types that we wish to use, and the framework will resolve them at runtime. What this means, is that rather than using the new keyword and passing arguments to constructors, we will call some special methods from our dependency injection library.

So in our StockValuer example we would write something like:

var container = new DependencyInjectionContainer();

container.Register<IInterestRateProvider, InterestRateProvider>();
container.Register<INewsFeedReader, NewsFeedReader>();
container.Register<IHistoricPriceReader, HistoricPriceReader>();

container.Register<IStockValuer, StockValuer>();

Then, when the stock valuer is used in your real code, say, a function like,

double GetPortfolioValue(Stocks[] stocks, IStockValuer stockValuer)
{
...
}

The dependency injection framework will create all these types at run time, and provide them to the method. We have to explicitly provide the very bottom level arguments, things like the configuration, but the framework resolves everything else.

Why is it bad?

I think this is a pretty awful way to program. Here are my reasons why.

It Makes Our Code Hard to Read and Understand

One of the guiding principles behind dependency injection is that it doesn’t matter what specific implementation of the IThingy interface you get, just that it is an IThingy. This is all very well and good in principle, but not in practice. Whenever I am reading or debugging code, and I want to know what it actually does, I always need to know what specific implementation of IThingy I am dealing with. What’s even worse, is that DI frameworks break IDEs. Because various types and constructors are resolved at runtime, semantic search no longer works. I can’t even look up where a type is created anymore!

It Encourages Us to Write Bad Code

Dependency injection frameworks encourage us to write our code as a mish-mash of dependent classes without any clear logic. Each individual piece of real working code gets split out into it’s own class and divorced from it’s actual context. We end up with a bewildering collection of types that have no real world meaning, and a completely baffling dependency graph. Everything is now hidden behind an interface, and every interface has only one implementation.

It turns Compile Time Errors into Runtime Errors

For me this is an absolutely unforgivable cardinal sin. Normally, in a statically typed language, if you don’t provide the right arguments to a method this is a compile time problem. In fact normally this is something that is picked up by your IDE before you even try to build your code. Not so with dependency injection frameworks. Now you will not discover if you have provided the correct arguments until you run your code!

To give an example, I was working on a web app that was built using dependency injection. One day we merged some changes, built and tested our code, and deployed it to the test environment. While it was running, it crashed. We had forgotten to register one of the arguments a new method was using, and, the dependency injection framework couldn’t resolve this at runtime. This is something we could have easily spotted if we were writing our code without dependency injection magic. Instead our type error was only discovered when a specific end point of our service was hit.

It is Outrageously Verbose

The sort of code we write with DI frameworks is naturally very verbose, lots of interfaces and lots of classes. I once stripped castle windsor out of a C# project and halved the number of lines without changing the functionality at all. The problem with really verbose code is that it is harder it is to maintain. Indeed the more lines of code you have, the more bugs you will have.

Worse than this though, is the tests. There is a solution of sorts to the issue with runtime errors mentioned above. We write unit tests to validate the type correctness of our code. This to me is a pretty mad solution. It only catches the type errors that you remember to test for and it bloats our code base even more.

It is Far Too Complicated.

Using a DI framework requires us to learn an entirely new meta-langauge that sits on top of C# or Java. We need to understand all kinds of tricks and gotchas. Instead of building up programs with a few basic tools, we are now writing them with a complex unintuitive meta language that describes how to evaluate dependency graphs at runtime. Dependency injection takes something simple but inelegant, a factory, and turns it into a incomprehensible mess.

What Level of Object Orientation are you?

I’ve seen a lot of arguments, both in real life and on the internet, about object orientation. One of the problems with these arguments is that people don’t always agree on what object orientation actually is. In my mind, the phrase has three distinct meanings. I don’t claim that any of what follows is the strict academic definition, it is just what I think people are usually talking about when they say “object oriented”.

Simply Using Classes and Methods

Sometimes when people say object orientation, they really just mean using classes and methods. Classes allow us to define composite types and to define functions on those types which we call methods. This isn’t really object orientation. You’ll find user defined composite types in plenty of languages that aren’t object oriented like Haskell. It is nice to be able to define a function on your new type as a method. However the fact that methods can be called only with the appropriate object is really just static typing.

Mutating State

The property that really characterises object orientation for me is mutable state. Object orientation doesn’t really make sense without mutability. Objects allow us to expose an API and abstract away the underlying implementation. This means that the objects we create contain some inner state and expose methods that use and potentially change this data. Many functional languages discourage mutability. For example in F# all variables are immutable by default and you have to use the mutable keyword to make them mutable.

Inheritance and Polymorphism

The final flavour of object orientation is the deepest and the darkest. This is the use of inheritance and polymorphism. This really is unambiguously object orientation by anyone’s definition. Real, old school object orientation will involve vast and complicated hierarchies of classes. You will have to deal with an advanced taxonomy of user defined types.

Inheritance does not only allow for simple code re-use. It allows you to take advantage of polymorphism. This means that at run-time different methods will be called depending on the type of your variable. So you might define a virtual method in your abstract base class, and have a different implementation for that method in each of the subclasses.

This sort of stuff, complex inheritance hierarchies and runtime polymorphism has fallen out of fashion but you will still see a lot of it in enterprise software, particularly at large financial institutions.

Don’t Get Caught out by Covariance!

Downcasting is bad, you shouldn’t do it, it’s code smell and it’s an anti-pattern. Unfortunately, in real world code you will see plenty of it. So you need to be aware of the pottential pitfalls of using downcasting. One that caught me out recently is covariance.

In C++, you can quite freely cast between types. You can take a pointer to some memory and cast it to any type you like, and read out what you get. Generally speaking you won’t get anything useful. If you’re not careful you’ll get “undefined behaviour”. If you have data stored in memory, the assumption is that you know that type that data is, and you are responsible for using it appropriately.

In languages like Java and C#, things are different. Here, the runtime checks our casts and will throw an exception if it thinks you have gone wrong. The consequences of this difference can sometimes be surprising.

Let’s look at an example. Suppose we have a base class Security defined like so:

public class Security
{
    int Id;
    public Security(int id)
    {
        Id = id;
    }
}

and two subclasses, Stock:

public class Stock : Security
{
    string Name;
    public Stock(int id, string name) : base(id)
    {
        Name = name;
    }
}

and Bond:

public class Bond : Security
{
    double Rate;
    public Bond(string name, double rate) : base(id)
    {
        Rate = rate;
    }
}

Now if we have a reference of type Security, that actually points to a Stock, we can happily downcast it to a reference of type Stock, like this:

Security s1 = new Stock(1, "GOOG");
Stock AsStock = (Stock)s1;
Console.WriteLine(s1.Name);

Because, Stock is a subtype of Security we can cast a reference to a Stock to a reference to a Security. This works because every Stock will have all the fields of a Security, in the same relative locations in memory.

However, if you have a reference of type Security that points to a Security or a Bond, and try and cast it to a Stock, you’ll have trouble. If we run the following code

Security s1 = new Bond(1, 0.02);
Stock AsStock = (Stock)s1;
Console.WriteLine(s1.Name);

we will see a run time exception of the form:

Unhandled exception. System.InvalidCastException: Unable to cast object of type 'Casting.Bond' to type 'Casting.Stock'.

This makes sense, the runtime knows that the object we have a reference to is not a Stock. It knows that we can’t cast this object to a stock in a sensible way. So the runtime stops us by throwing an exception.

Let’s look at a slightly more complicated example. Suppose, instead of a single object, we had a whole array of them. The following code:

Security[] securities = new Stock[]{new Stock(1, "ABC"), new Stock(2, "DEF")};
Stock[] stocks = (Stock[]) securities; 
Console.WriteLine(stocks[0].Name);

will run happily. We can cast a reference of type Security[] that points to a Stock[], to a reference of type Stock[]. However if we try

Security[] securities = new Security[]{new Stock(1, "ABC"), new Stock(2, "DEF")};
Stock[] stocks = (Stock[]) securities; 
Console.WriteLine(stocks[0].Name);

we will get an InvalidCastException:

Unhandled exception. System.InvalidCastException: Unable to cast object of type 'Casting.Security[]' to type 'Casting.Stock[]'.

This might seem a little surprising. The objects in our array are still actually Stocks. We know that we can cast a reference of type Security that points to a Stock to a reference of type Security. Why can’t we cast the Security[] reference to a Stock[] reference?

It’s a subtle one. When we cast an array reference, we are not casting the objects in the array, we are casting the array itself. So in the first array example, we are casting a reference of type Security[] to a reference of type Stock[]. The runtime knows that the reference actually points to an object of type Stock[], so this is fine. There will only ever be Stock objects in this array. Even though we have a reference of type Security[] pointing to this array, we can’t do something like:

Security[] securities = new Stock[]{new Stock(1, "ABC"), new Stock(2, "DEF")};
securities[0] = new Bond(2, 0.02);

the runtime knows that our array is of type Stock[], to it throws an exception:

Unhandled exception. System.ArrayTypeMismatchException: Attempted to access an element as a type incompatible with the array.

However, in the second example, we have a reference of type Security[] that points to an array of type Security[]. Although this array only contains stocks, the runtime cannot in general say whether that is true or not. Suppose we had done something like this instead:

Security[] securities = new Security[]{new Stock(1, "ABC"), new Stock(2, "DEF")};
securities[0] = new Bond(2, 0.02);
Stock[] stocks = (Stock[]) securities; 

We can of course add a Bond to an array of type Security[], the runtime doesn’t keep track of all this, which is why it has no way of knowing if the securities array really does contain objects of type Stock or something else.

The name of this type of casting is Covariance. Eric Lippert, one of the original designers of C#, has a pretty good blog about it. The really important point is that when we have an array of some base type, even if we are able to downcast the individual members of that Array, the runtime might stop us from downcasting the entire array.

This tripped me up in my day job last week. I was performing a data load that returned an array of type Security[], I knew this array contained only objects of type Stock and so I cast it to Stock[]. I merged this into master, but then our regression tests failed. Thankfully my mistake was caught before making it into prod.

Don’t Use Objects in Python – part 3

So my previous blog posts, were posted on reddit and I got a lot of interesting feedback. I’ve decided to address the issues that were brought up all together in one place, and this is it! I’ve grouped all the different ideas that came up into their own sections below.

This is against community guidelines

Some people were upset that what I recommended was against the community guidelines. They’re correct, it is. However that isn’t really an argument against doing something. Community guidelines and standards aren’t some holy scripture that must always be obeyed. The entire point of my post was that the common usage of object orientation of Python is bad, I doubt there is anyway I could say that without also advocating against the normal Python standards. There is certainly some value in educating people about community standards, but the value of an idea cannot be reduced to how compliant it is with Python coding guidelines.

Dictionaries are Actually Objects

So a couple of people pointed out that internal to python there is inheritance, and that datatypes like Dictionary actually inherit from the Python Object type. Sure, that’s fine, I don’t have a problem with how the internals of Python implement the various data structures. Indeed the internals are overwhelmingly written in C, so it is not at all relevant, when we’re talking about actual Python code.

Objects are Abstractions of Python Dictionaries

Some commenters argued that Objects in Python are abstractions sitting on-top of dictionaries. This just isn’t true. An abstraction hides the details below it. For example, variables are an abstraction that hide the details of how registers and memory access actually work. Because they limit what you see and what you can do, they make using memory a lot easier. Python objects don’t hide any of the details of a dictionary, they are just syntactic sugar, all the operations of a normal python dictionary are available right their on the object with slightly different syntax, you can even access the dictionary itself with the __dict__ attribute. So this isn’t really true.

It’s all just an implementation Detail

So a few people took issue with the fact that I was talking about the specifics of how objects and classes are actually implemented, rather than talking about them in an abstract sense. I think this has missed the point. I am not quibbling with how objects are implemented. My point is that objects don’t give any extra functionality beyond using dictionaries. I brought up the implementation as a way to demonstrate this. My point doesn’t rest on the fact that there is an actual dictionary internal to objects. It rests on the fact that objects are really just a collection of attributes with string names, these attributes can be added, deleted and modified at run time, and that there really isn’t anything more of value in the Python Object. The fact that this collection of attributes is implemented by a dictionary specifically isn’t important.

In particular some people stressed that what I was talking about was just an implementation detail unique to CPython, and that somehow it is different in some other Python implementation. I can’t stress this enough, it doesn’t matter what the actual implementation of the attributes of an object are, it is still bad. But also, no-one actually uses PyPy, Jython or IronPython. They are basically irrelevant.

Having methods defined on types is Convenient

I think this was probably the best point raised. I concede that if you want to maintain some state, and mutate it at runtime, it is convenient to wrap that state in an object and have specific methods defined on that object. The alternative is to just store your state using some built in types and operate on it with functions. However, what it seems you really want to do, restrict functions to only work with specific types, is to use static types. This seems like a roundabout way to achieve that. If you want static typing, you’re not going to be happy with Python. Really what you are doing here, is associating certain bits of data and functionality, like in an object oriented language, but without any guarantee of what the data actually is.

Python Objects are Different if you use __slots__

A lot of people brought up __slots__. Yes, if you use the __slots__ attribute Python objects work differently and my criticism doesn’t apply exactly. But, by default, python objects don’t use __slots__. The default implementation is actually pretty important, because it’s the one that is going to get used most often. In fact I’ve never seen someone use __slots__, I’ve only ever come across it as an obscure trick that gets recommended for python experts. It’s not a robust defence of a language feature to say, “actually there’s this special thing you can do that totally changes how it is implemented”. That’s really a sign that the default implementation is not good.

So __slots__ does change how objects work, and makes them a lot less like a special wrapper around a simple dictionary. But I don’t think it resolves any of the fundamental problems with using objects in Python. They are basically just a collection of attributes, we cannot say anything about them apart from how many attributes there are and what their names are. In particular they don’t specify the shape of the data.

One benefit of __slots__ is that your objects will use memory more efficiently. I would say, however, that you shouldn’t be worrying about memory optimisations like this in a high level language. Also, if there are memory optimisations in your language they shouldn’t be controlled in an opaque unintuitive way like the __slots__ attribute.

Don’t Use Objects in Python – part 2

In my previous post I explained that objects in python are really just special wrappers for dictionaries. There is one dictionary that contains the attributes of the object and another dictionary that contains the attributes of the class. Indeed we can really just think of the __init__ method as a static method that adds elements to a dictionary.

Objects in Python do however have one feature that you cannot get by just passing around dictionaries: inheritance.

Let’s suppose we are using some Arbitrary Object-Oriented Language. This language looks and works a lot like C#, Java or C++. We define two classes, Security and Bond in this language. Security has a single field, an integer, called Id:

public class Security
{
   public int Id;
   public Security(int id)
   {
        Id = id;
   }
}

Bond inherits from Security and adds another field, this time a double called Rate:

public class Bond : Security
{
    public double Rate;
    public Bond(int id, double rate) : super(id)
    {
        Rate = rate;
    }
}

What do we get by making Bond a subclass of Security? The most obvious thing we get is that Bond will get a copy of the implementation inside Security. So in this example, as Security has a public field called Id, Bond has one as well. We can think of Security as a user defined type that is built up out of primitive types. By using inheritance Bond extends this type.

In Python we can do something similar. First we define our two classes:

class Security:
    def __init__(self, id):
        self.id = id

and:

class Bond(Security):
    def __init__(self, id, rate):
        self.rate = rate
        Security.__init__(self, id)

Now, this time when we define Bond as a subclass of Security what do we get? Python objects are not composite types like in our object oriented language. In Python objects are really just dictionaries of attributes and these attributes are only distinguished by their names. So our bond class could have a numeric value in it’s Rate attribute, but it could also have a string value, or a list or any other type. When we subclass in python we are not extending a user defined type, we are just re-using some attribute names.

In our Arbitrary Object Oriented language, there is another advantage to making Bond a subclass of Security: the ability to treat Bond as a Security. To understand what this means, suppose we have a function that prints the ids of a list of Securities:

static void printPortfolio(Security[] securities)
{
    string ids = "";
    foreach(Security security in securities)
    {
        ids += (" " + security.id);
    }
    Console.WriteLine(ids);
}

Now, this function specifies it’s single parameter must be an array of Securities. However, by the magic of polymorphism, we can actually pass in an array of types that inherit from Security. When they are passed in they are automatically cast to type Security. This can be pretty useful, in particular it makes it a lot easier to reason about our code.

Now let’s define the same function in Python:

def print_portfolio(securities):
    ids = ""
    for security in securities:
        ids += (" " + str(security.id))
    return ids

On the face of it, this is very similar. However we are not really using polymorphism in the same way as we are in our object oriented language. We could actually pass a list of any objects into the print_portfolio function, and, as long as they had a id attribute, this would execute happily. We could, for example, define a completely unrelated class like so:

class Book:
    def __init__(self, id):
        self.id = id

and pass a list of these into our print_portfolio function without any problems. Indeed in Python we can dynamically add attributes to an object, so we could even create an empty class:

class Empty:
    def __init__(self):
        pass

and assign a id attribute to it at runtime, via:

e = Empty()
e.id = "Hello"

and then enclose it in a list [e] and pass it into the print_portfolio function.

There’s one more thing we get with inheritance in an object oriented language: access to protected members. When we mark a method or field as protected it will only be accessible from within objects of that type or types that inherit from it. However in Python there are no protected methods or fields, everything is public.

So there are three reasons why I don’t think inheritance makes sense in Python:

  • Python classes aren’t really composite types, so it doesn’t make sense to extend them
  • Inheritance doesn’t give us extra access to protected methods, as everything in python is public anyway
  • Inheritance in Python doesn’t give us the benefit of polymorphism because in Python there are really no restrictions on what objects we pass around

So, that is why there really is no extra benefit to using an object in Python rather than a dictionary.