When working with NUnit in MonoDevelop, I ran into a bit of weirdness where rebuilds within Unity would delete MonoDevelop’s reference to the nunit.framework dll. The eventual solution was to import the dll directly into the root of the Unity Assets folder. On Windows the dll was located at “Program Files (x86)/Unity/MonoDevelop/Addins/NUnit/”.
You can use NUnit, just likely not to test code that directly depends on UnityEngine.dll - that dll is mostly a shell to act as an interface for the actual engine. You could consider running Unity as a headless server and communicate to it via an API of your own, testing the results of the operations, but that seems more trouble than it's worth.
The approach I use is to remove as much work as possible to classes that do not depend on UnityEngine at all, abstracting the actual system work. For instance, your AI can probably be implemented mostly independent of the engine's functions, with the colliders, raycasting and whatnot acting mostly as a sensory mechanism. You then write unit testing for these standalone classes, and use them on your MonoBehaviors as well.
I'm thinking your only nice option would be to have your unit test solution launch and build your project (via editor batch mode command-line parameters) and then run the built standalone - using Debug.Log and the log files for print/expect.
Listing your previous attempts is always a good idea though. Maybe you were onto something that just needs a bit of tweaking.
It is possible to unit test the vast majority of your code using NUnit outside of Unity from monodevelop, as sought by the original poster.
As Ricardo states this does mean changing the way you depend on UnityEngine.dll. See our post on the matter and a framework that looks like UnityEngine.dll but is testable.
In terms of a monodevelop project structure, the key is to keep your tests in an entirely separate MonoDevelop solution (outside of the Unity project folder) that references the MonoDevelop projects that Unity creates automatically (Assembly-CSharp-vs, etc).
This way, Unity still creates all your MonoDevelop projects for you but won’t obliterate your test project.
In Dec 2013, Unity Released the Unity Test Tools which has both a NUnit runner but also a Integration Test Runner for running Test Scenarios inside of Unity. While you can run the Unit Test Suite Runner in Unity proper you can also just open up MonoDevelop and the tests with the MonoDevelop NUnit runner.