As far as I'm aware, there's absolutely nothing within the open-source world that compares with Kayako. Sorry, but Kayako is a beast of a program, and it'd be quite the undertaking for an open-source project.
Kayako is a beast. We just rolled onto a real license for MSM - I wouldn't imagine it any other way. Only because of the fact that Kayako is ... a fucking beast.
However if you're looking for something that's free and even somewhat close... I see a lot of companies using osTicket:: Open Source Support Ticket System which IMO is a piece of shit.