'That,' 'this page,' and 'that one' are different: 1 win and 3 losses by treating pronouns uniformly
Voice‑search systems often lose context when users ask follow‑up questions like “Can you explain that in more detail?” A study showed that simply appending the last user utterance to the query worsened results in most cases. The solution was to add the entire prior exchange—user question plus avata…
When a voice‑enabled avatar answers a user’s question, the next turn often contains a pronoun that points back to the previous answer. In Japanese, pronouns like “that” and “this” carry different meanings depending on whether they refer to the current screen or a prior utterance. A recent experiment on a Japanese tech site revealed that simply appending the user’s last utterance to the search query actually hurt performance in three out of four test cases. The fix? Include the full prior exchange—both the user’s question and the avatar’s answer—when rewriting the query.
What Went Wrong With the Naïve Fix?
The initial approach was straightforward: when a user said, “Can you explain that in more detail?” the system would tack the previous utterance onto the search query. The idea was that the query would now contain enough context to retrieve the correct article. However, the data told a different story. On a real site with 38 pages, the baseline score for that utterance was 0.435, while the modified score jumped to 0.602—still incorrect. The system retrieved the homepage instead of the article the avatar had just discussed.
In another case, the user asked, “What other articles are on this site?” The baseline score was 0.489 and the modified score dropped to 0.429, again missing the target. Only one out of four tests improved, showing that the naïve addition of the last utterance was not the right strategy.
Understanding Deictic Pronouns in Voice‑Screen Interfaces
Japanese grammar distinguishes between situational deixis—pronouns that refer to something present in the current situation—and textual deixis—pronouns that refer to something mentioned earlier in the conversation. In a voice‑screen UI, the “situation” is the screen itself. Thus, “this page” refers to the current screen, while “that” usually points back to the avatar’s previous answer.
Three types of textual deixis emerged from the logs:
- Immediate textual deixis – pronouns that refer to the immediately preceding exchange (e.g., “that” after an answer).
- Distant textual deixis – pronouns that refer to an interaction several turns ago (e.g., “earlier about avatar”).
- Non‑deictic interrogatives – questions that stand alone and do not rely on prior context (e.g., “Why is silent streaming necessary?”).
Why the Full Exchange Matters
When the system rewrote the query by adding the entire previous exchange—user question plus avatar answer—it achieved the correct ranking in all cases. For example, the query “Can you explain that in more detail?” combined with the avatar’s earlier explanation produced a score of 0.741, placing the right article at the top. This result demonstrates that the pronoun “that” was correctly interpreted as referring to the avatar’s prior answer, not to the user’s original question.
In contrast, simply appending the user’s utterance or the previous avatar response separately failed to capture the full context. The system either retrieved unrelated articles or missed the target entirely.
Handling Distant Deixis with Sliding Windows
For pronouns like “earlier about avatar,” a sliding‑window approach that adds the last few exchanges proved ineffective. Adding one or two turns back still yielded unrelated results because intervening topics diluted the relevance. Only when the system searched for the specific keyword “avatar” within the conversation history and included that relevant exchange did the correct article surface.
This finding suggests that for distant deixis, it is better to retrieve the most relevant prior exchange rather than blindly adding more context. A simple 2‑gram character overlap search for the keyword sufficed, avoiding the need for complex morphological analysis.
Practical Implications for Voice‑Search Design
Conversational query rewriting is essential for handling anaphora and ellipsis in voice interactions. However, rewriting every query can degrade performance if the query already contains enough information. The recommended strategy is to default to no rewriting in stable domains and use search confidence as a trigger for selective rewriting.
In voice‑screen interfaces, developers should recognize that pronouns like “this” can refer to the screen itself. Therefore, query rewriting logic must differentiate between situational and textual deixis to avoid misinterpretation.
What Happens Next?
Future work will explore automated detection of when to rewrite a query based on utterance length and content. The goal is to maintain low latency while ensuring high retrieval accuracy across a wide range of conversational contexts.
Key Takeaways
- Adding only the last user utterance to a search query can worsen results.
- Including the full prior exchange—user question plus avatar answer—restores accuracy for pronouns like “that.”
- Distant deixis requires targeted retrieval of the relevant exchange, not a blanket sliding window.
- Not all queries need rewriting; selective rewriting based on confidence scores is preferable.
- In voice‑screen UIs, “this” often refers to the screen, not the conversation.
Frequently Asked Questions
- Q: Why does “that” sometimes refer to the avatar’s answer?
A: In Japanese, “that” is a textual deictic pronoun that points back to the most recent utterance in the conversation. - Q: Can I use the same strategy for English voice assistants?
A: The principle applies, but English pronouns may behave differently; careful linguistic analysis is needed. - Q: Does adding context increase response time?
A: No, the rewriting step can be performed by a lightweight model in under 100 ms, keeping latency minimal. - Q: How do I detect when to rewrite?
A: Use utterance length and keyword presence; if the query is short or lacks explicit terms, consider rewriting. - Q: What if the user’s question is ambiguous?
A: The system can ask for clarification or use the full prior exchange to disambiguate.
Why it matters
Accurate pronoun interpretation is vital for natural voice interactions; without it, users receive irrelevant results and lose trust in the assistant.
Key points
- Naïve query rewriting can hurt performance
- Full prior exchange improves accuracy
- Distant deixis needs targeted retrieval
- Selective rewriting based on confidence is best
- Voice‑screen interfaces treat ‘this’ as the screen
Frequently asked questions
Why does ‘that’ sometimes refer to the avatar’s answer?
In Japanese, ‘that’ is a textual deictic pronoun that points back to the most recent utterance in the conversation.
Can I use the same strategy for English voice assistants?
The principle applies, but English pronouns may behave differently; careful linguistic analysis is needed.
Does adding context increase response time?
No, the rewriting step can be performed by a lightweight model in under 100 ms, keeping latency minimal.
How do I detect when to rewrite?
Use utterance length and keyword presence; if the query is short or lacks explicit terms, consider rewriting.
What if the user’s question is ambiguous?
The system can ask for clarification or use the full prior exchange to disambiguate.
