I am trying to make rotate an object relative to its camera.
I think quaternion angles are the way to go but nothing I have tried so far works.

basically depending on how the camera looks at an object (from top, at 45 degree,or from side, or back or anything in between) I want to rotate it left/right or up/down depending but not in local or global space but relatively to the camera. so if camera is looking at model at at a 45 degree angle i wish to make it rotate around both y and z axis ( depending on that angle). (but this also means looking at it from the back or the front will have another change in axis)

I just don’t know how i would go about it.

Edit:
Adding image of what i would like to know how to do:

After you have no idea how much trial and error and learning Quaternions i have finally figured out how to do this rotation.

Thanks to everyone that has posted an answer but the answers are not what I needed. As far as i understood them all answers were for a stationary camera, or to move the camera around. I needed to rotate an object relative to a dynamically changing camera.

THis is my solution (my rotate function)

public void Rotate(float RotateLeftRight, float RotateUpDown)
{
float angleX = Camera.mainCamera.transform.localEulerAngles.x;
float angleY = Camera.mainCamera.transform.localEulerAngles.y;
//Debug.Log(angleX + " , " + angleY);
//Camera's angle at z is not needed.
float percentAngleX = (float)angleX / 90.0f;
float percentAngleY = 0.0f;
float x = 0.0f;
float y = 0.0f;
float z = 0.0f;
if(angleY < 90)
{
//Only worry about x and z for up down
percentAngleY = angleY /90.0f;
x += RotateUpDown * (1 - percentAngleY);
z += RotateUpDown * percentAngleY;
y += RotateLeftRight * (1- percentAngleX);
x += RotateLeftRight * (percentAngleX) * (percentAngleY);
z += RotateLeftRight * (percentAngleX) * (1 - percentAngleY);
}
else if(angleY < 180)
{
percentAngleY = (angleY - 90.0f) / 90.0f;
z += RotateUpDown * (1 - percentAngleY);
x -= RotateUpDown * (percentAngleY);
y += RotateLeftRight * (1- percentAngleX);
z -= RotateLeftRight * (percentAngleX) * (percentAngleY);
x += RotateLeftRight * (percentAngleX) * (1 - percentAngleY);
}
else if(angleY < 270)
{
percentAngleY = (angleY - 180.0f) / 90.0f;
x -= RotateUpDown * (1 - percentAngleY);
z -= RotateUpDown * (percentAngleY);
y += RotateLeftRight * (1- percentAngleX);
x -= RotateLeftRight * (percentAngleX) * (percentAngleY);
z -= RotateLeftRight * (percentAngleX) * (1 - percentAngleY);
}
else if(angleY < 360)
{
percentAngleY = (angleY - 270.0f) / 90.0f;
z -= RotateUpDown * (1 - percentAngleY);
x += RotateUpDown * (percentAngleY);
y += RotateLeftRight * (1- percentAngleX);
z -= RotateLeftRight * (percentAngleX) * (percentAngleY);
x += RotateLeftRight * (percentAngleX) * (1 - percentAngleY);
}
Vector3 relativeUp = transform.InverseTransformDirection(Vector3.down);
Vector3 relativeRight = transform.InverseTransformDirection(Vector3.left);
Vector3 relativeForeward = transform.InverseTransformDirection(Vector3.forward);
rotateBy = Quaternion.AngleAxis(z, relativeForeward)
* Quaternion.AngleAxis(y , relativeUp)
* Quaternion.AngleAxis(x , relativeRight);
newDeltaObtained = true;
}
Then in my update function update my rotation
void Update()
{
if(newDeltaObtained)
{
this.gameObject.transform.localRotation *= rotateBy;
newDeltaObtained = false;
}
}

This code allows me to rotate the object with relative to the camera regardless of where the camera is pointing and at what angle it is pointing there. It took me a long time to figure this out (over 21 hours) so i hope other looking for something like this can no see it.

New rotate code here:

public void Rotate(float rotateLeftRight, float rotateUpDown, bool isPlayer)
{
useUpdate = false;
//Unsure of how much below code changes outcome.
float sensitivity = 0;
if (isPlayer)
{
sensitivity = .5f;
}
else
{
sensitivity = .25f;
}
//Get Main camera in Use.
Camera cam = Camera.mainCamera;
//Gets the world vector space for cameras up vector
Vector3 relativeUp = cam.transform.TransformDirection(Vector3.up);
//Gets world vector for space cameras right vector
Vector3 relativeRight = cam.transform.TransformDirection(Vector3.right);
//Turns relativeUp vector from world to objects local space
Vector3 objectRelativeUp = transform.InverseTransformDirection(relativeUp);
//Turns relativeRight vector from world to object local space
Vector3 objectRelaviveRight = transform.InverseTransformDirection(relativeRight);
rotateBy = Quaternion.AngleAxis(rotateLeftRight / gameObject.transform.localScale.x * sensitivity, objectRelativeUp)
* Quaternion.AngleAxis(-rotateUpDown / gameObject.transform.localScale.x * sensitivity, objectRelaviveRight);
newDeltaObtained = true;
}

Inspired by @Scroodge 's approach, I got to this solution.

This script goes on the camera.
All you need to do is insert/switch/adapt your swipe input to the ‘Rotate’ part, the last line on the code below(and remove the ‘Input.GetAxis’ that is just for testing input).

Also, you just need to call the MovedCamera() function whenever you reposition your camera and is about to use the swipe function, so you update the rotation of ‘axis’ and is ready to rotate the object properly.
Remember that the ‘axis’ object is positioned at (0,0,0). Moving it or having the object not positioned on the center, will give you wrong rotations.

var axis:Transform;
var target:Transform;
function MovedCamera(){
target.parent=null;
axis.LookAt(transform);
target.parent=axis;
}
function Update () {
if(Input.GetKeyDown("space"))MovedCamera();
axis.Rotate(Vector3.up*Time.deltaTime*Input.GetAxis("Horizontal"), Space.Self);
}