Unfortunately the ObjectSelector (the editor window you’re talking about) is an internal class, so it can’t be used directly. It might be possible to show it with reflection, but it’s not guaranteed that it will work in the future.
The ObjectField function does this when you click on the icon:
ObjectSelector.get.Show(obj, objType, property, allowSceneObjects);
But since the whole ObjectSelector class is declared as internal, the class is unknown to the scripting environment.
If i find the time i can write a wrapper function to show it, however, it might be a bit more complicated, since they also assign the controlid that opened the selector to distinguish between multiple instances i guess.
edit
btw, the Show function looks like this:
public void Show(Object obj, Type requiredType, SerializedProperty property, bool allowSceneObjects)
{
So the function doesn’t return anything. It directly sets the selected object in the SerializedProperty you have to pass to the function.
second edit
Here i’ve written some wrappers for “some” functions properties that might be useful. It’s all a bit hacky, but it works 
The big problem is to detect when the selector is closed. You can poll the isvisible property to detect it has been closed. However after it has been closed you can’t get the selected object because the selector doesn’t exist anymore, so you have to poll the selected object while it’s visible. The function “HasJustBeenClosed” returns true only one time when the selector is closed.
Here’s a simple example:
// In an EditorWindow:
Texture texture;
Texture selection;
void OnGUI()
{
if(GUILayout.Button(texture))
{
ObjectSelectorWrapper.ShowSelector(typeof(Texture));
}
}
void Update()
{
if(ObjectSelectorWrapper.isVisible)
{
selection = ObjectSelectorWrapper.GetSelectedObject<Texture>();
}
if(ObjectSelectorWrapper.HasJustBeenClosed())
{
texture = selection;
}
}
This example will show a button, when you click the button the object selector should show up and when it’s closed the button should show the selected texture. You can of course directly use the “selection”, however you might need to detect a selection change and repaint your EditorWindow or the change will not be visible until the window is repainted. This example has not been tested. You might need another way to call Update, maybe the update delegate.
Here’s the actual wrapper:
using UnityEngine;
using UnityEditor;
using System.Collections;
using System.Reflection;
public static class ObjectSelectorWrapper
{
private static System.Type T;
private static bool oldState = false;
static ObjectSelectorWrapper()
{
T = System.Type.GetType("UnityEditor.ObjectSelector,UnityEditor");
}
private static EditorWindow Get()
{
PropertyInfo P = T.GetProperty("get", BindingFlags.Public | BindingFlags.Static);
return P.GetValue(null,null) as EditorWindow;
}
public static void ShowSelector(System.Type aRequiredType)
{
MethodInfo ShowMethod = T.GetMethod("Show",BindingFlags.Public | BindingFlags.Instance | BindingFlags.DeclaredOnly);
ShowMethod.Invoke(Get (), new object[]{null,aRequiredType,null, true});
}
public static T GetSelectedObject<T>() where T : UnityEngine.Object
{
MethodInfo GetCurrentObjectMethod = T.GetMethod("GetCurrentObject",BindingFlags.Static | BindingFlags.Public);
return GetCurrentObjectMethod.Invoke(null,null) as T;
}
public static bool isVisible
{
get
{
PropertyInfo P = T.GetProperty("isVisible", BindingFlags.Public | BindingFlags.Static);
return (bool)P.GetValue(null,null);
}
}
public static bool HasJustBeenClosed()
{
bool visible = isVisible;
if (visible != oldState && visible == false)
{
oldState = false;
return true;
}
oldState = visible;
return false;
}
}