OpenAI says ChatGPT can now browse the internet to provide users with 'current and authoritative information' even as many major news sites have blocked its data scraping bot
. Without access to these sites, what would the effect be on the quality of ChatGPT’s answers?
The new function to browse the internet, and how ChatGPT interacts with sites depends on how sites themselves configure the crawler. OpenAI said, “Updates [for the browsing function] include following robots.txt and identifying user agents so sites can control how ChatGPT interacts with them.” Robots.txt is a document that site owners can configure to tell crawlers which parts of a website they can access. Putting in instructions on the document to block the ChatGPT crawler can forbid it to access and scrape either the entire site or just parts of it.
The browsing function was originally rolled out in May but was disabled in July as users found out that it could. OpenAI said then, “We’ve learned that ChatGPT’s ‘Browse’ beta can occasionally display content in ways we don’t want, e.g. if a user specifically asks for a URL’s full text, it may inadvertently fulfill this request. We are disabling Browse while we fix this – want to do right by content owners.
The revived browse function will be again available first to paying Plus and Enterprise, and Microsoft Bing users, but will be expanded to all users “soon.” new voice and image features, allowing users to converse via voice with ChatGPT, or ask it about a particular image a user took with a camera.