Function calling metadata missing in streamed chunk response from LMStudio (qwen2.5-7b-instruct-1m)

[juliarvalenti]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

// server.ts snippet

const model = new ChatOpenAI({
  // modelName: "deepseek-r1-distill-qwen-7b",
  modelName: "qwen2.5-7b-instruct-1m",
  openAIApiKey: "lmstudio", // Placeholder
  configuration: {
    baseURL: "http://localhost:1234/v1"
  },
  streaming: true,
});

app.post("/stream-text", async (req: Request, res: Response) => {
  try {
    const { messages } = req.body;

    const chatMessages = [
      new SystemMessage(SYSTEM_PROMPT),
      ...messages.map((m: any) =>
        m.role === "user"
          ? new HumanMessage(m.content)
          : new AIMessage(m.content)
      ),
    ];

    const tools = [
      startNewGameTool,
      playMoveTool,
      drawArrowTool,
      calculatorTool,
    ];
    const boundModel = model.bindTools(tools);

    const streamIterator = await boundModel.stream(chatMessages);
    for await (const chunk of streamIterator) {
      if (!final) {
        final = chunk;
      } else {
        final = final.concat(chunk);
      }
      // **I'm checking this chunk here for the appropriate function calling metadata and not seeing it**
      console.dir(chunk, { depth: null });
    }
    console.log("Final streaming result:", final);
    res.end();
  } catch (error) {
    console.error("Error in streaming:", error);
    res.end();
  }
});

// tools.ts

import { DynamicStructuredTool } from "@langchain/core/tools";
import { z } from "zod";

export const startNewGameTool = new DynamicStructuredTool({
  name: "start_new_game",
  description: "Clears the board and starts a new game.",
  schema: z.object({}),
  func: async () => {
    return "Starting a new game now.";
  },
});

export const playMoveTool = new DynamicStructuredTool({
  name: "play_move",
  description:
    "Places a stone on the board at a given position for a given color.",
  schema: z.object({
    color: z.enum(["black", "white"]).describe("Color of the stone."),
    position: z
      .tuple([z.number(), z.number()])
      .describe("Coordinates [x, y] of the move."),
  }),
  func: async ({ color, position }) => {
    return `Playing a ${color} stone at position ${position}`;
  },
});

export const drawArrowTool = new DynamicStructuredTool({
  name: "draw_arrow",
  description: "Draws an arrow between two points on the board.",
  schema: z.object({
    start: z.tuple([z.number(), z.number()]).describe("Start position [x, y]."),
    end: z.tuple([z.number(), z.number()]).describe("End position [x, y]."),
    color: z.string().describe("Color of the arrow."),
  }),
  func: async ({ start, end, color }) => {
    return `Drawing an arrow from ${start} to ${end} in ${color}.`;
  },
});

export const calculatorTool = new DynamicStructuredTool({
  name: "calculator",
  description: "Can perform mathematical operations.",
  schema: z.object({
    operation: z
      .enum(["add", "subtract", "multiply", "divide"])
      .describe("The type of operation to execute."),
    number1: z.number().describe("The first number to operate on."),
    number2: z.number().describe("The second number to operate on."),
  }),
  func: async ({ operation, number1, number2 }) => {
    // Functions must return strings
    if (operation === "add") {
      return `${number1 + number2}`;
    } else if (operation === "subtract") {
      return `${number1 - number2}`;
    } else if (operation === "multiply") {
      return `${number1 * number2}`;
    } else if (operation === "divide") {
      return `${number1 / number2}`;
    } else {
      throw new Error("Invalid operation.");
    }
  },
});

Description

I am running qwen2.5-7b-instruct-1m via LMStudio and streaming responses to my frontend using an Express server.

Behavior

This setup was working fine until I added function calling support. Now, the streamed response does not include the expected function call metadata. I'm using the below to dive into and examine the data coming from the stream

    const streamIterator = await boundModel.stream(chatMessages);
    for await (const chunk of streamIterator) {
      console.dir(chunk, { depth: null });
    }

If none of the messages in the conversation explicitly invoke a function call (For example simply saying "Hello"), the response is correctly streamed as AIMessageChunk objects and the above code logs this:

AIMessageChunk {
  lc_serializable: true,
  lc_kwargs: {
    content: "'.",
    tool_call_chunks: [],
    additional_kwargs: {},
    id: 'chatcmpl-fqaxt6eb8e4e3dkx269d6',
    response_metadata: { usage: {} },
    tool_calls: [],
    invalid_tool_calls: [],
    usage_metadata: undefined
  },
  lc_namespace: [ 'langchain_core', 'messages' ],
  content: "'.",
  name: undefined,
  additional_kwargs: {},
  response_metadata: { prompt: 0, completion: 0, usage: {} },
  id: 'chatcmpl-fqaxt6eb8e4e3dkx269d6',
  tool_calls: [],
  invalid_tool_calls: [],
  tool_call_chunks: [],
  usage_metadata: undefined
}

This is good because I can see tool_calls, tool_call_chunks, and invalid_tool_calls and parse all of that data appropriately.

However, if I prompt the model to explicitly call a function (e.g., "What is 5+5" for calculatorTool or "Move a black stone to G5" via the playMoveTool), then stream only outputs ChatMessageChunk objects that lack the function call metadata. Additionally, it doesn't respond with any content.

ChatMessageChunk {
  lc_serializable: true,
  lc_kwargs: {
    content: '',
    role: undefined,
    response_metadata: { usage: {} },
    additional_kwargs: {}
  },
  lc_namespace: [ 'langchain_core', 'messages' ],
  content: '',
  name: undefined,
  additional_kwargs: {},
  response_metadata: {
    prompt: 0,
    completion: 0,
    finish_reason: 'stop',
    system_fingerprint: 'qwen2.5-7b-instruct-1m',
    model_name: 'qwen2.5-7b-instruct-1m',
    usage: {}
  },
  id: undefined,
  role: undefined
}

Expected behavior

When the model decides to call a function, the streamed response should contain tool_calls or tool_call_chunks metadata so I can parse and execute the function calls.

Observations

• I can see in LMStudio logs that the model is attempting to stream a response.
  However, the streamIterator in my Express server only receives ChatMessageChunk objects, which do not contain the function call details.

Question

Is there something wrong with how I am handling streaming function calls? Should boundModel.stream(chatMessages) be expected to always return function call metadata in the streamed chunks?

Additionally, it is entirely possible that this is an issue for lmstudio or the qwen model instead of Langchain. Please advise if you can! Thank you.

System Info

[email protected] | MIT | deps: 12 | versions: 310
Typescript bindings for langchain
https://github.com/langchain-ai/langchainjs/tree/main/langchain/

keywords: llm, ai, gpt3, chain, prompt, prompt engineering, chatgpt, machine learning, ml, openai, embeddings, vectorstores

dist
.tarball: https://registry.npmjs.org/langchain/-/langchain-0.3.15.tgz
.shasum: c6c8b17bf20c870795e31515f48abde7d13ccc9d
.integrity: sha512-+DQ4I2iy4b5sErkxo6jAkgmumvhgqLwLB2fmiGl3yDt8+VVZdB1MUULZMzf+6ubarNc7Mwn/sxHUqK4GhEndhg==
.unpackedSize: 2.9 MB

dependencies:
@langchain/openai: >=0.1.0 <0.5.0 js-tiktoken: ^1.0.12 jsonpointer: ^5.0.1 openapi-types: ^12.1.3 uuid: ^10.0.0 zod-to-json-schema: ^3.22.3
@langchain/textsplitters: >=0.0.0 <0.2.0 js-yaml: ^4.1.0 langsmith: >=0.2.8 <0.4.0 p-retry: 4 yaml: ^2.2.1 zod: ^3.22.4

maintainers:

• nfcampos [email protected]
• jacoblee93 [email protected]
• andrewnguonly [email protected]
• davidduong [email protected]
• hwchase17 [email protected]
• basproul [email protected]

dist-tags:
latest: 0.3.15 next: 0.3.2-rc.0 tag-for-publishing-older-releases: 0.2.20

published 2 weeks ago by jacoblee93 [email protected]

---

Platform: Windows
Node: 20.17.0
Npm: 10.8.2

[dosubot[bot]]

Hello @juliarvalenti! I'm a bot designed to help you with bug fixes, questions, and contributions. I'll be here to assist you while you wait for a human maintainer.

When handling streamed responses in LangChain, it's important to ensure that the function call metadata is correctly processed and included in the streamed chunks. Here are a few things to check and consider:

1. Tool Call Handling: Ensure that your stream handling code is set up to detect and process tool calls. When a tool call is detected in the stream, the corresponding tool should be invoked, and the response should be processed and included in the output. This involves checking for tool_calls in the stream chunks and handling them appropriately ^[1].

2. Streamed Response Structure: The expected structure of the streamed response when a function call is made should include AIMessage with tool_calls, followed by a ToolMessage for each tool call. Ensure that your implementation adheres to this structure to avoid missing metadata ^[1].

3. Model and Tool Configuration: Verify that the model and tools are correctly configured to support function calls. The bindTools method should be used to associate the tools with the model, enabling it to utilize these tools during streaming ^[2].

4. Debugging and Logging: Use logging to debug the stream processing. Check if the model is indeed generating tool calls and if these are being correctly captured and processed in your server code. The absence of function call metadata might indicate an issue with how the stream is being handled or how the model is configured.

5. Model-Specific Behavior: Consider that the issue might be specific to the model or the LMStudio setup. Ensure that the model you are using supports function calls and that there are no limitations or specific configurations required for this feature.

By ensuring that these aspects are correctly implemented, you should be able to receive and process function call metadata in your streamed responses. If the issue persists, it might be worth investigating further into the model's capabilities or any specific configurations required by LMStudio.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[juliarvalenti]

@dosu

Actually, do you think this issue is more relevant for the maintainers of LMStudio instead? Is the protocol such that the role metadata is missing and that LMStudio should be appending it appropriately but isn't?

[dosubot[bot]]

This is a tough nut to crack, could really use your help on this, @jacoblee93.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[juliarvalenti]

Hi Jacob,

Sorry for the direct ping above. I'm working my way through this now. This may just be an issue with LMStudio.

[juliarvalenti]

@jacoblee93

I have replaced LMStudio with OpenAI directly and I'm not observing the behavior, so I don't think this is a langchain issue. I've opened up a ticket on LMStudio regarding this.

lmstudio-ai/lmstudio-bug-tracker#445