Summary Buffer Memory¶

The same CassandraChatMessageHistory can power a more sophisticated kind of chat memory that automatically maintains a summary of the past interactions, so as to provide a memory that virtually extends arbitrarily far back in the past while fitting a finite amount of tokens.

Notice¶

this demo is currently not working with the Azure OpenAI LLMs. Please run it with another LLM provider.

In [1]:

from langchain.memory import CassandraChatMessageHistory
from langchain.memory import ConversationSummaryBufferMemory
from langchain.chains import ConversationChain

from langchain.memory import CassandraChatMessageHistory from langchain.memory import ConversationSummaryBufferMemory from langchain.chains import ConversationChain

In [2]:

from cqlsession import getCQLSession, getCQLKeyspace
cqlMode = 'astra_db' # 'astra_db'/'local'
session = getCQLSession(mode=cqlMode)
keyspace = getCQLKeyspace(mode=cqlMode)

from cqlsession import getCQLSession, getCQLKeyspace cqlMode = 'astra_db' # 'astra_db'/'local' session = getCQLSession(mode=cqlMode) keyspace = getCQLKeyspace(mode=cqlMode)

In [3]:

message_history = CassandraChatMessageHistory(
    session_id='summarized-conversation-4321',
    session=session,
    keyspace=keyspace,
    ttl_seconds = 3600,
)

message_history = CassandraChatMessageHistory( session_id='summarized-conversation-4321', session=session, keyspace=keyspace, ttl_seconds = 3600, )

In [4]:

message_history.clear()

message_history.clear()

Below is the logic to instantiate the LLM of choice. We chose to leave it in the notebooks for clarity.

In [5]:

import os
from llm_choice import suggestLLMProvider

llmProvider = suggestLLMProvider()
# (Alternatively set llmProvider to 'GCP_VertexAI', 'OpenAI', 'Azure_OpenAI' ... manually if you have credentials)

if llmProvider == 'GCP_VertexAI':
    from langchain.llms import VertexAI
    llm = VertexAI()
    print('LLM from Vertex AI')
elif llmProvider == 'OpenAI':
    os.environ['OPENAI_API_TYPE'] = 'open_ai'
    from langchain.llms import OpenAI
    llm = OpenAI()
    print('LLM from OpenAI')
elif llmProvider == 'Azure_OpenAI':
    os.environ['OPENAI_API_TYPE'] = 'azure'
    os.environ['OPENAI_API_VERSION'] = os.environ['AZURE_OPENAI_API_VERSION']
    os.environ['OPENAI_API_BASE'] = os.environ['AZURE_OPENAI_API_BASE']
    os.environ['OPENAI_API_KEY'] = os.environ['AZURE_OPENAI_API_KEY']
    from langchain.llms import AzureOpenAI
    llm = AzureOpenAI(temperature=0, model_name=os.environ['AZURE_OPENAI_LLM_MODEL'],
                      engine=os.environ['AZURE_OPENAI_LLM_DEPLOYMENT'])
    print('LLM from Azure OpenAI')
else:
    raise ValueError('Unknown LLM provider.')

import os from llm_choice import suggestLLMProvider llmProvider = suggestLLMProvider() # (Alternatively set llmProvider to 'GCP_VertexAI', 'OpenAI', 'Azure_OpenAI' ... manually if you have credentials) if llmProvider == 'GCP_VertexAI': from langchain.llms import VertexAI llm = VertexAI() print('LLM from Vertex AI') elif llmProvider == 'OpenAI': os.environ['OPENAI_API_TYPE'] = 'open_ai' from langchain.llms import OpenAI llm = OpenAI() print('LLM from OpenAI') elif llmProvider == 'Azure_OpenAI': os.environ['OPENAI_API_TYPE'] = 'azure' os.environ['OPENAI_API_VERSION'] = os.environ['AZURE_OPENAI_API_VERSION'] os.environ['OPENAI_API_BASE'] = os.environ['AZURE_OPENAI_API_BASE'] os.environ['OPENAI_API_KEY'] = os.environ['AZURE_OPENAI_API_KEY'] from langchain.llms import AzureOpenAI llm = AzureOpenAI(temperature=0, model_name=os.environ['AZURE_OPENAI_LLM_MODEL'], engine=os.environ['AZURE_OPENAI_LLM_DEPLOYMENT']) print('LLM from Azure OpenAI') else: raise ValueError('Unknown LLM provider.')

LLM from OpenAI

Use the memory with summary in a Chain¶

In [6]:

memory = ConversationSummaryBufferMemory(
    llm=llm,
    chat_memory=message_history,
    max_token_limit=180,
)

memory = ConversationSummaryBufferMemory( llm=llm, chat_memory=message_history, max_token_limit=180, )

In [7]:

summaryConversation = ConversationChain(
    llm=llm, 
    memory=memory,
    verbose=True
)

summaryConversation = ConversationChain( llm=llm, memory=memory, verbose=True )

With the verbose output you can track the summary. It starts empty and once few messages have been exchanged it starts to "incorporate" those messages that would have been lost in the past.

In [8]:

summaryConversation.predict(input='Tell me about William Tell, the hero')

summaryConversation.predict(input='Tell me about William Tell, the hero')

> Entering new  chain...
Prompt after formatting:
The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.

Current conversation:

Human: Tell me about William Tell, the hero
AI:

> Finished chain.

Out[8]:

" William Tell was a legendary Swiss hero who lived in the late 13th or early 14th century. He is most known for his feat of shooting an apple off of his son's head with a crossbow. This act was said to be an act of defiance against a tyrannical Austrian ruler, Gessler, who had placed his hat atop a pole in the town square and demanded that all citizens bow to it. Tell's refusal to do so earned him his place in Swiss folklore."

In [9]:

summaryConversation.predict(
    input="Any relation to the apple from Cupertino?"
)

summaryConversation.predict( input="Any relation to the apple from Cupertino?" )

> Entering new  chain...
Prompt after formatting:
The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.

Current conversation:
Human: Tell me about William Tell, the hero
AI:  William Tell was a legendary Swiss hero who lived in the late 13th or early 14th century. He is most known for his feat of shooting an apple off of his son's head with a crossbow. This act was said to be an act of defiance against a tyrannical Austrian ruler, Gessler, who had placed his hat atop a pole in the town square and demanded that all citizens bow to it. Tell's refusal to do so earned him his place in Swiss folklore.
Human: Any relation to the apple from Cupertino?
AI:

> Finished chain.

Out[9]:

'I do not know.'

No messages have slided past the "recent window" yet -- correspondingly, the summary buffer is still empty:

In [10]:

memory.moving_summary_buffer

memory.moving_summary_buffer

Out[10]:

''

Continue the conversation - you might start to see the summary of past exchanges making its way to the final prompt for the LLM in the form of a "System" contribution to the Current conversation:

In [11]:

summaryConversation.predict(
    input="I heard the two apples may be in fact the very same fruit."
)

summaryConversation.predict( input="I heard the two apples may be in fact the very same fruit." )

> Entering new  chain...
Prompt after formatting:
The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.

Current conversation:
Human: Tell me about William Tell, the hero
AI:  William Tell was a legendary Swiss hero who lived in the late 13th or early 14th century. He is most known for his feat of shooting an apple off of his son's head with a crossbow. This act was said to be an act of defiance against a tyrannical Austrian ruler, Gessler, who had placed his hat atop a pole in the town square and demanded that all citizens bow to it. Tell's refusal to do so earned him his place in Swiss folklore.
Human: Any relation to the apple from Cupertino?
AI: I do not know.
Human: I heard the two apples may be in fact the very same fruit.
AI:

> Finished chain.

Out[11]:

" That seems highly unlikely given that William Tell's feat is said to have happened in the 13th or 14th century, while the Apple from Cupertino was founded in the late 20th century."

This is where the current summary buffer is stored

(Note: depending on the length of the messages so far, at this point you might or might not see an empty summary. In any case, keep reading and it will doubtless start to be nonempty.)

In [12]:

memory.moving_summary_buffer

memory.moving_summary_buffer

Out[12]:

'\nThe human asks the AI to tell them about William Tell, the hero.'

The whole conversation is still in the "recent" window. Below is what will start to happen in the ConversationSummaryBufferMemory at the next chat message: starting from the message list, and the summary so far (if any), the LLM is asked to come up with a new summary which will overwrite the current one, stored in memory.moving_summary_buffer.

In [13]:

print(memory.predict_new_summary(
    memory.chat_memory.messages,
    memory.moving_summary_buffer,
))

print(memory.predict_new_summary( memory.chat_memory.messages, memory.moving_summary_buffer, ))

The human asked the AI to tell them about William Tell, the hero. The AI explained that Tell was a legendary Swiss hero who was known for shooting an apple off of his son's head with a crossbow to defy a tyrannical Austrian ruler, Gessler. The human then asked if there was any relation to the Apple from Cupertino, to which the AI replied that it was unlikely given that Tell's feat happened centuries before the company was founded.

In [14]:

summaryConversation.predict(
    input='Ok, enough with apples. Let\'s talk starfish now.'
)

summaryConversation.predict( input='Ok, enough with apples. Let\'s talk starfish now.' )

> Entering new  chain...
Prompt after formatting:
The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.

Current conversation:
System: 
The human asks the AI to tell them about William Tell, the hero.
Human: Tell me about William Tell, the hero
AI:  William Tell was a legendary Swiss hero who lived in the late 13th or early 14th century. He is most known for his feat of shooting an apple off of his son's head with a crossbow. This act was said to be an act of defiance against a tyrannical Austrian ruler, Gessler, who had placed his hat atop a pole in the town square and demanded that all citizens bow to it. Tell's refusal to do so earned him his place in Swiss folklore.
Human: Any relation to the apple from Cupertino?
AI: I do not know.
Human: I heard the two apples may be in fact the very same fruit.
AI:  That seems highly unlikely given that William Tell's feat is said to have happened in the 13th or 14th century, while the Apple from Cupertino was founded in the late 20th century.
Human: Ok, enough with apples. Let's talk starfish now.
AI:

> Finished chain.

Out[14]:

" Sure! Starfish, or sea stars, are a group of echinoderms found in all of the world's oceans. They have five arms and a central disc, and they range in size from a few centimeters to over a meter in diameter. Starfish are found in a variety of habitats, from shallow tidal pools to deep sea floors. They are carnivorous predators and feed on mollusks, crustaceans, and other small organisms."

In [15]:

memory.moving_summary_buffer

memory.moving_summary_buffer

Out[15]:

"\nThe human asks the AI to tell them about William Tell, the hero. The AI explains that William Tell was a legendary Swiss hero from the late 13th or early 14th century, who is most known for shooting an apple off his son's head with a crossbow. This act was an act of defiance against an Austrian ruler, Gessler, who had placed his hat atop a pole in the town square and demanded that all citizens bow to it. The human then jokes about any relation to the apple from Cupertino."

Now the chat history has presumably reached a certain threshold and there is a summary. In the verbose output below, you will see it appearing in the prompt as a "System" role in the conversation recap.

In [16]:

summaryConversation.predict(
    input='I hear that some of those destroy marine ecosystems.'
)

summaryConversation.predict( input='I hear that some of those destroy marine ecosystems.' )

> Entering new  chain...
Prompt after formatting:
The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.

Current conversation:
System: 
The human asks the AI to tell them about William Tell, the hero. The AI explains that William Tell was a legendary Swiss hero from the late 13th or early 14th century, who is most known for shooting an apple off his son's head with a crossbow. This act was an act of defiance against an Austrian ruler, Gessler, who had placed his hat atop a pole in the town square and demanded that all citizens bow to it. The human then jokes about any relation to the apple from Cupertino.
Human: Tell me about William Tell, the hero
AI:  William Tell was a legendary Swiss hero who lived in the late 13th or early 14th century. He is most known for his feat of shooting an apple off of his son's head with a crossbow. This act was said to be an act of defiance against a tyrannical Austrian ruler, Gessler, who had placed his hat atop a pole in the town square and demanded that all citizens bow to it. Tell's refusal to do so earned him his place in Swiss folklore.
Human: Any relation to the apple from Cupertino?
AI: I do not know.
Human: I heard the two apples may be in fact the very same fruit.
AI:  That seems highly unlikely given that William Tell's feat is said to have happened in the 13th or 14th century, while the Apple from Cupertino was founded in the late 20th century.
Human: Ok, enough with apples. Let's talk starfish now.
AI:  Sure! Starfish, or sea stars, are a group of echinoderms found in all of the world's oceans. They have five arms and a central disc, and they range in size from a few centimeters to over a meter in diameter. Starfish are found in a variety of habitats, from shallow tidal pools to deep sea floors. They are carnivorous predators and feed on mollusks, crustaceans, and other small organisms.
Human: I hear that some of those destroy marine ecosystems.
AI:

> Finished chain.

Out[16]:

' Yes, that is true. Some species of starfish, notably the crown-of-thorns starfish, can be invasive and destructive to coral reef ecosystems. The starfish feed on the coral, often causing catastrophic damage to the reef.'

In [17]:

memory.moving_summary_buffer

memory.moving_summary_buffer

Out[17]:

"\nThe human asks the AI to tell them about William Tell, the hero. The AI explains that William Tell was a legendary Swiss hero from the late 13th or early 14th century, who is most known for shooting an apple off his son's head with a crossbow. This act was an act of defiance against an Austrian ruler, Gessler, who had placed his hat atop a pole in the town square and demanded that all citizens bow to it. The human then jokes about any relation to the apple from Cupertino, to which the AI responds that it is highly unlikely given the timeline of their respective histories."

From this point on, the summary will not grow indefinitely in size, but will rather try to cover a longer and longer conversation in a fixed amount of text.

In [18]:

print(memory.predict_new_summary(
    memory.chat_memory.messages,
    memory.moving_summary_buffer,
))

print(memory.predict_new_summary( memory.chat_memory.messages, memory.moving_summary_buffer, ))

The human asks the AI to tell them about William Tell, the hero. The AI explains that William Tell was a legendary Swiss hero from the late 13th or early 14th century, who is most known for shooting an apple off his son's head with a crossbow as an act of defiance against a tyrannical Austrian ruler. The human then jokes about any relation to the apple from Cupertino, to which the AI responds that it is highly unlikely given the timeline of their respective histories. The human then shifts the conversation to starfish, to which the AI explains that they are a group of echinoderms found in all the world's oceans. The AI also mentions that some species of starfish, such as the crown-of-thorns starfish, can be invasive and destructive to coral reef ecosystems.