Feature Request: Add cachePoint support for langchainjs aws lib

[tnlamda]

Checked

• I searched existing ideas and did not find a similar one
• I added a very descriptive title
• I've clearly described the feature request and motivation for it

Feature request

According to AWS Documentation, cache checkpoints are now supported by Claude 3.5 Haiku/Sonnet v2 as well as Amazon's Nova models. Cache checkpoints can be added to the system field for all of these models. However, langchain only supports strings for system messages, see here.

Motivation

My team has been given access to the new prompt caching feature, but we're unable to use it due to langchain's lack of support in the js library. A similar pull request has been made for adding cache checkpoints to the python lib.

Proposal (If applicable)

The system message validation needs to be updated to support the new cache checkpoint feature, then properly pass it along to Converse- I suspect it would already get passed along fine.

[AllenFang]

hi @tnlamda
you can take a look for this first. aws/aws-sdk-js-v3#6404

In short, likely cache check point only support in CLI and python lib, the aws-sdk-js-v3 doesn't support it yet. ETA is end of Q1 or early Q2.
ps (aws-sdk/client-bedrock-runtime is the underlying library that langchain js used for communicating to bedrock.

[tnlamda]

Hey @AllenFang thanks for the response. Q1/Q2 ETA is for opening up cache checkpoints publicly, it has been available in preview mode (to select users) since December. There is a private boto py SDK required if you're using the Converse API, but for the InvokeModel API there is no additional SDK requirement.

Just to test out the cachePoint feature, I removed the langchain implementation from my stack and replaced it with direct invokeModel calls to bedrock using @aws-sdk/client-bedrock-runtime (with no private SDK). I got this to work for both Anthropic (Sonnet/Haiku 3.5) models and Amazon Nova models. In the longterm I would like to use LangChain though, as it simplifies the more complicated interactions (tools, agents, etc).

Below is the requestBody for each of my calls to bedrock that were able to successfully add cache checkpoints.

This is how I call bedrock:

        const command = new InvokeModelCommand({
            accept: 'application/json',
            body: JSON.stringify(requestBody),
            contentType: 'application/json',
            modelId: modelId
        });
        return await bedrockClient.send(command);

requestBody for Anthropic's Claude (Sonnet/Haiku 3.5) Models is as follows:

{
    "anthropic_version": "bedrock-2023-05-31",
    "max_tokens": 1024,
    "system": "system query goes here",
    "messages": [
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "CACHE CHECKPOINT HERE",
                    "cache_control": {
                        "type": "ephemeral"
                    }
                },
                {
                    "type": "text",
                    "text": "user query goes here"
                }
            ]
        },
        {
            "role": "assistant",
            "content": [
                {
                    "type": "text",
                    "text": "```json"
                }
            ]
        }
    ],
    "temperature": 0,
    "stop_sequences": [
        "```"
    ]
}

And the requestBody for Amazon's Nova Pro/Lite models is:

{
    "system": [
        {
            "text": "system prompt goes here",
            "cachePoint": {
                "type": "default"
            }
        }
    ],
    "messages": [
        {
            "role": "user",
            "content": [
                {
                    "text": "users query goes here"
                }
            ]
        },
        {
            "role": "assistant",
            "content": [
                {
                    "text": "```json"
                }
            ]
        }
    ],
    "inferenceConfig": {
        "stopSequences": [
            "```"
        ]
    }
}

[AllenFang]

hi @tnlamda Today, langchain JS still use Converse API rather than InvokeModel. I check the langchain python lib, likely py support Converse as well as InvokeModel, that is why they can support cache checkpoint rapidly. I haven't tested the InvokeModel, but i think you are right based on your works on InvokeModel manner.

btw, i'm not a AWS guy so a noob question, what is different between Converse and InvokeModel? Can I say Converse is a higher order wrapper on InvokeModel and Converse like a stateful runtime approach?

Anyway, adding an InvokeModel approach into langchain js wouldn't be so hard I guess, I will see what I can do for it.

[tnlamda]

I'm fairly new to bedrock as well, but yeah you're correct, from my understanding Converse is essentially a wrapper around InvokeModel (you need invokeModel access to use Converse) that allows for easier interaction across numerous models, due to different models having different requestBody structures.

https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference.html

Sounds good let me know what you figure out!

[AllenFang]

@tnlamda trying to submit a draft #7810
Not sure whether or not team want to support InvokeModel, we will see.