Exclusive-Multiple AI companies bypassing web standard to scrape publisher sites, licensing firm says
-Multiple artificial intelligence companies are circumventing a common web standard used by publishers to block the scraping of their content for use in generative AI systems, content licensing startup TollBit has told publishers.
A Wired investigation published this week found Perplexity likely bypassing efforts to block its web crawler via the Robots Exclusion Protocol, or"robots.txt," a widely accepted standard meant to determine which parts of a site are allowed to be crawled.The News Media Alliance, a trade group representing more than 2,200 U.S.-based publishers, expressed concern about the impact that ignoring"do not crawl" signals could have on its members.
The company tracks AI traffic to the publishers' websites and uses analytics to help both sides settle on fees to be paid for the use of different types of content. "What this means in practical terms is that AI agents from multiple sources are opting to bypass the robots.txt protocol to retrieve content from sites," TollBit wrote."The more publisher logs we ingest, the more this pattern emerges."
The AI companies use the content both to train their algorithms and to generate summaries of real-time information.
United Kingdom Latest News, United Kingdom Headlines
Similar News:You can also read news stories similar to this one that we have collected from other news sources.
Man dies after being stabbed multiple times in Bolingbrook; 1 in custody, police sayA 23-year-old man has died after being stabbed multiple times in the south suburbs Tuesday night, police said.
Read more »
Cincinnati Reds Get Positive Injury News About Multiple Key PlayersFinally, some good injury news for the Cincinnati Reds
Read more »
Houston’s self-proclaimed ‘King of Bissonnet’ gets 40 years for trafficking multiple women, beating themA Houston man convicted of sex trafficking four women by means of force and of taking three women across state lines to engage in prostitution has learned his sentence.
Read more »
Google Meet’s adaptive audio connects mics from multiple laptops for clearer audioIf your conference room mic isn’t working, Google Meet can try syncing up the microphones and speakers of every laptop in the room.
Read more »
Central Pa. caretaker accused of neglecting woman found with multiple bruisesA supervisor asked the caretaker about the woman’s injuries on two occasions before she was taken to the hospital by family members, charging documents said.
Read more »
Riverside man accused of making multiple swatting calls, threatening to shoot up schoolsA Riverside man is facing several charges for allegedly making a series of swatting calls and threatening to commit mass violence at several schools.
Read more »