Slot Carryover

What is Slot Carryover?

Slot carryover is a central capability in modern AI-powered chatbots, especially within task-oriented dialogue systems. It allows the system to remember, reuse, and correctly apply previously gathered structured information (slots) across multiple conversational turns. In both spoken and text-based dialogue systems, slot carryover directly affects how well the system maintains context and responds to users in a coherent and helpful manner.

This is particularly important in complex, multi-turn, and multi-domain conversations where users refer to earlier stated preferences or entities, often using pronouns or implicit references. For example, if a user says, “Book a flight to Paris,” and then “Find me a hotel there,” the system must understand that “there” refers to “Paris” and carry over the relevant slot.

Effective slot carryover eliminates the need for users to repeat information, supports natural language reference resolution, and ensures that the chatbot maintains an accurate conversational state as the dialogue evolves.

Formal Definition:
Slot carryover is the task where a model makes a binary decision for each candidate slot from previous dialogue context, determining if it should be carried over to the current turn to support intent fulfillment.

This process is fundamental to dialogue state tracking (DST), as it involves tracking the emergence and evolution of slots and their values throughout the dialogue, mapping slots between potentially different schemas or domains, and applying learned models or rules to decide which slots are relevant for the current turn.

Core Technologies

Tracking

Monitoring the emergence and evolution of slots and their values throughout the dialogue.

Mapping

Translating slots between potentially different schemas or domains (e.g., mapping “WeatherLocation” in a weather app to “City” in a travel booking app).

Selection

Applying learned models or rules to decide which slots are relevant for the current turn.

Technical Challenges

Slot carryover, especially in real-world, multi-turn, and multi-domain dialogue systems, raises several technical challenges:

Modeling Approaches

Rule-Based Approaches

Early slot carryover implementations used hand-crafted rules, such as always carrying over the most recent slot, or applying heuristics based on slot recency and type.

Naive Baseline: Always carries over all slots from the immediate previous turn.

Rule Baseline: Employs hand-crafted rules for certain slot types or based on conversation patterns.

Limitations:
Rule-based systems are brittle and do not generalize well to unseen dialogue flows or new domains. They perform poorly in cases involving long-distance slot references or schema heterogeneity.

Neural Network Architectures

The state-of-the-art in slot carryover relies on neural models that can dynamically manage context and slot relevance:

Pointer Networks:
Allow the model to select and order slots from the dialogue history, capturing explicit references to earlier slots. They model the sequence of slots and their ordering, which is important when multiple slots may be referenced and their order matters.

Transformer-Based Models:
Use self-attention to model dependencies between slots and across dialogue turns. This enables the network to focus on which slots from the entire dialogue history are relevant to the current user turn, regardless of their position.

Attention Mechanisms:
Both word-level and stream-level attention mechanisms help the model focus on the most relevant utterances and slot mentions, improving the resolution of ambiguous or long-distance references.

Embedding-Based Schema Mapping:
By representing slot keys and values as embeddings, models can compute similarity between slots across heterogeneous schemas. This is especially important for mapping slots between domains with different naming conventions or structures.

End-to-End Carryover Decision:
Modern approaches frame slot carryover as a binary classification or selection task over a candidate set of slots, using contextual encodings, slot embeddings, and recency indicators.

Implementation Considerations

Slot Schema Mapping

Slot carryover across domains often requires mapping slot keys and values between different schemas.

Techniques:

Label Embeddings: Averaging pre-trained word embeddings for slot keys and values to compute similarity and candidate mappings.

Data-Driven Mapping: Learning mappings from data rather than relying on static dictionaries or hand-crafted rules.

Candidate Slot Generation

The system generates a candidate set of slots from the full conversation context, then transforms these into the schema used by the current domain. This step is critical to limit computation and focus carryover decisions on plausible slots.

Evaluation Metrics

Key performance metrics for slot carryover include:

Precision: Proportion of carried-over slots that are correct.

Recall: Proportion of relevant slots that are successfully carried over.

F1 Score: Harmonic mean of precision and recall.

Memory Management and Privacy

Memory Types

Privacy and Scalability

Data Retention: Strict policies for what information is stored and for how long.

User Consent: Mechanisms for opt-in/opt-out and transparency.

Secure Storage: Encryption and access controls for sensitive slot values.

Scalability: Efficient indexing and retrieval to support large user bases and long dialogue histories.

Privacy Risks and Protection:
Chatbots can inadvertently store sensitive user data (e.g., location, personal identifiers) in slot memories, raising privacy concerns. Best practices include:

• Limiting storage of personally identifiable information (PII)
• Providing users with explicit options to control what is remembered
• Regularly purging slot memories per privacy policy
• Using encrypted storage and access controls

Applications and Examples

Multi-Domain Assistants

Slot carryover is indispensable for assistants that support multiple domains (e.g., weather, local search, booking). It enables seamless transitions and natural reference resolution.

Example Dialogue:

Technical Dialogue Snippets

Single-Domain Carryover:

User: What's the weather in San Francisco?
Bot: It's sunny and 18°C.
User: Book a hotel there for tonight.

Carryover: “San Francisco” is transferred from weather to hotel booking.

Cross-Domain Schema Mapping:

User: Find Italian restaurants in Paris.
Bot: Here are some options.
User: Reserve a table at the first one.

Carryover: “Paris” is mapped from “Location” in search to “City” in booking.

Long-Distance Carryover:

User: I want to fly to Berlin.
Bot: What dates are you considering?
User: Next weekend.
Bot: Should I book a hotel there too?
User: Yes, please.

“Berlin” is referenced over several turns and domains.

Benchmarks and Datasets

Evaluating slot carryover models requires robust datasets that represent real-world conversational complexity:

DSTC (Dialog State Tracking Challenge) Series:

• DSTC2: Focused on restaurant booking, widely used for slot carryover and state tracking tasks
• DSTC8, DSTC9: Later versions introduce multi-domain and more challenging scenarios

Schema-Guided Dialogue (SGD) Dataset:

• Large-scale, multi-domain task-oriented dataset, designed to evaluate schema mapping and carryover across numerous services and domains

Hugging Face Dialogue State Tracking Datasets Collection:

• Curated collection including MultiWOZ, WOZ, and others

Amazon Alexa Internal Dataset:

• Used for evaluating slot carryover in production-like settings

Challenges and Limitations

Error Propagation: Mistakes in slot extraction or carryover can propagate, compounding downstream errors.

Schema Alignment: Automated mapping of slots across domains with disparate schemas remains complex, especially at scale.

Ambiguity Resolution: Implicit references, pronouns, and context-dependent expressions require sophisticated co-reference and context modeling.

Data Privacy: Storing and processing sensitive user data requires robust privacy safeguards, encryption, and compliance (e.g., GDPR).

Computational Cost: Transformer-based and attention-heavy models increase computational and memory requirements for large context windows.

Summary Comparison

References

• Amazon Science: Improving Long Distance Slot Carryover in Spoken Dialogue Systems
• arXiv: Improving Long Distance Slot Carryover in Spoken Dialogue Systems (Chen et al., 2019)
• ACL Anthology: Improving Long Distance Slot Carryover in Spoken Dialogue Systems
• Interspeech 2018: Contextual Slot Carryover for Disparate Schemas (Naik et al.)
• Hugging Face: Dialogue State Tracking Datasets Collection
• Mozilla Foundation: How to Protect Your Privacy from ChatGPT and Other AI Chatbots
• Tencent Cloud Techpedia: Memory Types
• Microsoft Research: Dialog State Tracking Challenge
• Hugging Face Datasets: Schema-Guided DSTC8