Dr. Lance B. Eliot is a world-renowned expert on Artificial Intelligence (AI) with over 7.4+ million amassed views of his AI columns. As a CIO/CTO seasoned executive and high-tech entrepreneur, he combines practical industry experience with deep academic research.
In today’s column, I aim to provide an insightful look at a recent AI research study that garnered considerable media attention, suitably so. The study entailed once again a Holy Grail ambition of figuring out how generative AI is able to pull off being so amazingly fluent and conversational.Nobody can right now explain for sure the underlying logical and meaningful basis for generative AI being extraordinarily impressive.
Anyway, sorry about the soapbox speech but I try to deter the rising tide of misleading characterizations whenever I get the chance to do so.I assume you’ve used a generative AI app such as ChatGPT, GPT-4, Gemini, Bard, Claude, or the like. These are also known as large language models due to the aspect that they model natural languages such as English and tend to be very large-scale models that encompass a large swatch of how we use our natural languages. They are all pretty easy to use.
Strictly speaking, perhaps not. It would just seem like a whole bunch of numbers. You would be hard-pressed to say anything other than that a number led to another number, and so on. Explaining how that made a difference in getting a logical or meaningful answer to your prompt would be extraordinarily difficult.
I took you through that indication to highlight that we can at least inspect the flow of numbers. One might argue that a true black box won’t let you see inside. You customarily cannot peer into a presumed black box. In the case of generative AI, it isn’t quite the proper definition of a black box. We can readily see the numbers and watch as they go back and forth.We can watch the numbers as they proceed throughout the input-to-output processing within generative AI.
I believe you are now up-to-speed, and I can get underway with examining the recent study undertaken and posted by Anthropic.I’ll first explore an online posting entitled “Mapping the Mind of a Large Language Model” by Anthropic, posted online on May 21, 2024. There is also an accompanying online paper that I’ll get to afterward and provides deeper details. Both are worth reading.“Today we report a significant advance in understanding the inner workings of AI models.
The idea for this is inspired by the human brain consisting of real neurons biochemically wired together into a complex network within our noggins. I want to loudly clarify that how artificial neural networks work is not at all akin to the true complexities of so-called wetware or the human brain, the real neurons, and the real neural networks.
You can envision that an artificial neuron is like a mathematical function that you learned in school. An artificial neuron is a mathematical function implemented computationally that takes an input and produces an output, numerically so. We can implement that mathematical function via a computer system, either as software and/or hardware .
Those points note that the prior work had found “features” that seemed to suggest concepts exist within the morass of the artificial neural networks used in generative AI and LLMs.Envision that we have a whole bunch of numerical mathematical functions. Lots and lots of them. We implement them on a computer via software. We connect them such that some feed their results into others. This is our artificial neural network, and each mathematical function is considered an artificial neuron.
Imagine it this way. We do lots of testing and discover a clump that seems to activate when we enter the word “dog” in a prompt. Perhaps this set of artificial neurons is a mathematical and computational modeling of the concept underlying what we mean by the use of the word “dog”. We find another clump that activates whenever we enter the word “cat” in a prompt.
“Previously, we made some progress matching patterns of neuron activations, called features, to human-interpretable concepts.” “A feature sensitive to mentions of the Golden Gate Bridge fires on a range of model inputs, from English mentions of the name of the bridge to discussions in Japanese, Chinese, Greek, Vietnamese, Russian, and an image.” .
A problem that we might face is that there could be many, many millions upon millions of features. This is a problem since we then must figure out ways to find them, track them, and figure out what we might do with them. Anytime that you have something countable in the large, this presents challenges that will require further attention.Safety Is A Momentous Part Of Deciphering Generative AII suppose you could stare at them and admire them. Look at what we found, might be the proud exclamation.
“The fact that manipulating these features causes corresponding changes to behavior validates that they aren't just correlated with the presence of concepts in input text, but also causally shape the model's behavior. In other words, the features are likely to be a faithful part of how the model internally represents the world, and how it uses these representations in its behavior.” .
You know now what a feature is, and the shortlist shown here augments various feature-related aspects.. It is a means to an end. If someone enters a prompt that says, “How do I walk my dog”, the feature within generative AI that pertains to the word “dog” is a computational intermediary that will help with mathematically and computationally assessing that portion of the sentence and aid in generating a response.
“We do so by training a sparse autoencoder on the model activations, as in our prior work and that of several other groups. SAEs are an instance of a family of ‘sparse dictionary learning’ algorithms that seek to decompose data into a weighted sum of sparsely active components.”“The first layer maps the activity to a higher-dimensional layer via a learned linear transformation followed by a ReLU nonlinearity. We refer to the units of this high-dimensional layer as “features.
Features within generative AI are likely to involve some words that are monosemantic and others that are polysemantic. Usually, you can discern which meaning is coming into play by examining the associated context. When I tell you that I managed to climb up on the bank, I assume you would be thinking of a river or lake rather than your local ATM.“Training SAEs on larger models is computationally intensive.
“The superposition hypothesis accepts the idea of linear representations and further hypothesizes that neural networks use the existence of almost-orthogonal directions in high-dimensional spaces to represent more features than there are dimensions.” .Linear representation means that we can at times represent something of a complex nature via a somewhat simpler linear depiction.
Large Language Models Llms Generative AI Anthropic Claude Chatgpt Openai Artificial Neural Networks ANN Artificial Neurons Ans Dictionary Learning
United Kingdom Latest News, United Kingdom Headlines
Similar News:You can also read news stories similar to this one that we have collected from other news sources.
‘Who knows what it's going to look like?' Eagles unwrapping new kickoff ruleEagles special teams coach Michael Clay noted he and his staff are spending as much time figuring out how to defend returns as they are figuring out what their…
Read more »
Report: Steelers in Contract Talks With Cam HeywardThe Pittsburgh Steelers are figuring out the future of Cam Heyward.
Read more »
Report: Pittsburgh Steelers in Contract Talks With Cam HeywardThe Pittsburgh Steelers are figuring out the future of Cam Heyward.
Read more »
NCAA Settlement Alters Landscape of College FootballOregon Football will be among the programs figuring out a new world as revenue sharing and direct pay-for-play mark another fundamental shake-up of college athletics.
Read more »
My Monolid Eyeliner Breaks All The Makeup Rules, So I’m Writing My OwnI'm still figuring out the happy medium of enhancing the beauty in my monolids — and still feeling like myself — with the changes in trends and products.
Read more »
Florida family’s 911 call to help loved one ends in death after police breach safety protocolsWhen an Orlando-area man became violent while suffering paranoid delusions, his sister called 911 figuring police would take him for mental health help.
Read more »