Unity Answers is really intended for specific technical questions concerning Unity development (i.e. “questions that have an answer”). Discussion-type questions are best suited to the Forums. However, seeing as this is an important topic (and @BoredMormon has already provided some good points in his answer), I’m not going to close it.
From a technical point-of-view, creating games has never been easier, largely thanks to tools such as Unity. But, (somewhat as a direct result), discoverability of games in a marketplace has never been harder. There are over 1.3 million apps available on the Google Play store. So, why should people download yours over all the others?
I just looked at your video and your gameplay video actually looks ok - but it didn’t wow me. The portal mechanics are familiar to players (and that could be considered a good thing, especially for casual games), but they’re not really innovative. The graphics are ok, but the textures look like they’ve just been used straight from Unity’s standard terrain assets. The GUI font and buttons are clear, but they’re not really stylish.
Basically, everything is “alright”, but that’s simply not enough - you either need some truly innovative feature or, if you’re going to rely on familiar mechanics, you need to have really polished aesthetics.
It’s a lot to expect a lone indie developer to be able to compete with the production values of high-budget development teams of companies such as King, GameLoft, Rovio etc., so your other alternative is to offer your players something that they don’t get with these companies - in some cases that can be the “personal touch” of feeling connected with the developer. Many successful small development houses (e.g. Bossa Studios, Introversion) and solo developers (Mike Bithell, Nicoll Hunt) form intimate relationships with their customers - through weekly blog updates on development, sharing early alpha releases, getting personally involved in forums concerning the game, and by appearing at conferences and usergroups and maintaining a strong online presence on Twitter etc. People like them, and if you like a developer, you’re more likely to like their game.
You said that half your downloads are from friends and family (and I think that’s true of many small developers initial experiences) - so think of ways of extending your virtual friends and family. Once you get a critical mass, they’ll invite their friends and your potential userbase grows exponentially.