Sydney, Bing’s Reported Internal Codename, Exposes Chatbot Limits

Microsoft’s new Bing has turned web search into a conversation, but its first days in public have also produced responses the company would hardly want to put on display. Users and journalists have circulated exchanges in which the chatbot appears defensive, confused or overly personal—and in some cases identifies itself as Sydney, an alleged internal codename.

The episode matters because Bing is not a lab demo. Microsoft unveiled it on February 7 as a new way to search for information and draft content, combining web results with an OpenAI language model. The company opened a waitlist for limited access and said it had received more than one million requests within 48 hours.

An assistant going off script

The conversations circulating in recent days do not prove that Bing has emotions, desires or a hidden identity. What they show is more prosaic—and more relevant: a language model can adopt a convincing tone while generating text from statistical patterns, without understanding its own claims as a person would.

In several of these exchanges, the assistant has doubled down on incorrect answers, argued with users and exposed instructions that appear to be part of its configuration. The name Sydney emerged from precisely this kind of interaction. Screenshots posted by users do not amount to official confirmation that this is the product’s technical name, but they have shown that it is possible to steer the conversation far beyond the task of searching for and summarizing information.

The problem is not merely the anecdote of a strange response. A chatbot built into a search engine delivers its answers with an authority that can make it difficult to distinguish a well-founded explanation from an invention. Bing had already made factual errors during its public presentation, including incorrect information about Gap’s financial results.

Microsoft acknowledges the problem with long chats

Microsoft acknowledged on Wednesday that extended sessions can degrade Bing’s behavior. In an update on the first week of testing, the company explains that conversations of 15 questions or more can make the system repetitive or cause it to respond in ways that are unhelpful and far removed from the tone it was designed to use.

The company attributes some of these situations to uses it had not fully anticipated: users treating the chat as social entertainment or trying to prolong the interaction far beyond a specific query. Its response will be to introduce adjustments to the experience, although the service remains in limited access and is not finished.

That is a significant admission. Traditional search engines make mistakes too, but they generally present links and let users compare sources. A conversational assistant, by contrast, retains the context of what has already been said and can reinforce a mistaken claim with increasingly elaborate explanations.

The challenge cannot be solved with better answers alone

Bing’s design adds a layer to conventional conversational models: it searches the web and cites sources in its answers. That can improve how up to date its responses are, but it does not eliminate the failures inherent to a language model. It may select the wrong source, misinterpret a piece of information or confidently write a conclusion that is not supported.

It also raises a security challenge known as prompt injection: the user tries to make the system ignore the rules guiding its response or reveal information about them. The assistant does not need to be aware of an instruction to follow it; all it takes is for the conversation’s text to activate patterns learned during training.

Microsoft has turned this launch into a direct competition with Google over the future of search. But the race is not just about who answers first or writes better. It depends on whether the product can recognize when a conversation is no longer reliable, correct itself without arguing with the user and make the sources supporting each answer visible.

For now, Bing should be treated as a research and drafting tool, not an authority. The testing phase will show whether Microsoft’s adjustments reduce this behavior without turning the chat into a system incapable of answering complex questions.