Add custom functions for Muse to call

Does Muse Chat support me providing a set of functions for it to be able to call when it deems relevant?

If not, is this planned?
If yes, can you provide an example?

Hey @Statement - thank you for reaching out!
Would you mind elaborating what type of functions you’d like to call? :slight_smile: Would they be unity functions or API functions? Any insight here would be very helpful.

Currently we provide function calling between tooling within the Muse suite such as opening Muse Texture from having asked about creating a texture in Muse Chat. Hoping to extend that to other valuable tools in the Unity ecosystem, where relevant.

Cheers, Martina and the Muse Chat team

I’d like to register functions that Muse could use, like I can with ChatGPT.

A facade entry would be something like Muse.AddPlugin(myObject), which have methods on it, possibly decorated with attributes to tell muse which methods to load into the callable set of functions via reflection. I am working on something on my own pet project that does this. It would allow people to easily integrate their automation scripts with natural language. I admit the exact usecase is vague, but it sounds cool! :smiley:

But I mean if you are doing something similar I might as well just give up all this serialization logic with ChatGPT.

I don’t know how you interface with your LLM, but if it’s like OpenAI, there’s a web endpoint that expects a json with functions defined in a schema.

        public class DocClass
        {
            /// <summary>
            /// Field summary.
            /// </summary>
            public int test;
        }

        /// <summary>
        /// A documented method.
        /// </summary>
        /// <param name="dummyParameter">A documented parameter.</param>
        static void DocumentedMethod_ArrayOfDocClass(DocClass[] dummyParameter) { }

        [TestMethod]
        public void ToJson_WithDocumentedFunctionParameters_ArrayOfDocClass_ReturnsExpectedSchema()
        {
            // Arrange
            var expectedJson = @"
            {
                ""type"": ""function"",
                ""function"": {
                    ""name"": ""DocumentedMethod_ArrayOfDocClass"",
                    ""description"": ""A documented method."",
                    ""parameters"": {
                        ""type"": ""object"",
                        ""properties"": {
                            ""dummyParameter"": {
                                ""type"": ""array"",
                                ""items"": {
                                    ""type"": ""object"",
                                    ""properties"": {
                                        ""test"": { 
                                            ""type"": ""integer"",
                                            ""description"": ""Field summary.""
                                        }
                                    }
                                },  
                                ""description"": ""A documented parameter.""                
                            }
                        },
                        ""required"": [ ""dummyParameter"" ],
                        ""additionalProperties"": false
                    }
                }
            }";

            // Act
            var actualJson = generator?.ToJson(nameof(DocumentedMethod_ArrayOfDocClass), DocumentedMethod_ArrayOfDocClass);

            // Assert
            AssertJsonAreEqual(expectedJson, actualJson);
        }

Like I am doing this with my tool call generator. It uses reflection from a source Delegate type, and looks up suitable documentation. Right now it uses the xml documentation from a build, but you could just as well change the documentation provider to read from attributes.

Anyway, all of that is then sent to the LLM, which returns with a suitable tool to call. It would be nice with some approval like “do you want to run this” and politely say, I very much do, and then I dunno, I just find the tech so cool. :smiley:

Like, if someone made a trello plugin, or whatever, it could be cool to be able to chat about their trello board through muse via the plugin bindings that asset developers could set up.