Acuracy Metrics on Training Google AI

Google Maps tips & tricks
1

We have a post from Ermest marked as the answer to the initial inquiry and are honing in on the epicenter of this topic @PaulPavlinovich , @ErmesT and to make it easier for other guides to follow here is a reset of the post thread:

HOW IS BEST TO UNDERSTAND THE PROGRESS PATHWAY OF THE EDUCATION OF THE GOOGLE ARTIFICIAL INTELLIGENCE FROM THE STANDPOINT OF ACCURACY IN DECISION MAKING?

Is there an accessible metrics showing where the AI is at in this developmental curve?

Let me frame this conversation thread with a key detail - we arrived to this aspect by discussing the spam filter’s level of correctness in making determinations in application of the posting rules.

When a report of a review or profile is made, the AI makes decisions and performs actions / inactions. The initial thread started with a look at a scenario where a business owner had posted a rating to boost the company rank under pressure of accumulating negative reviews. My focus was on how the Algorithmic Artificial Intelligence can determine investigatory necessities to “reason” through the scenario to a decision. Paul and Ermest you have both clarified that the rules should be followed and the report made, which I accept and support. From interest in understanding what we are doing as Local Guides who do report wrong data when we find it, the next question is WHAT SENSE DO WE HAVE OF THE AI’S ACCURACY?

The moral / ethics aspect is just one part of the picture as this is only one avenue of evaluation of correctness. Like you explain Paul, Telsa Auto Pilot crashes far less than human drivers but it does crash and the quality of ability of an AI will depend on how it was made and then trained. Naturally the training process results are monitored and adjusted for optimized results.

Accountability for correctness of AI decisions requires tracking of the effectiveness of the training process - or Grading Learning.

Millions of us experience impacts of a AI decisions - for example, a business owner told me last week that a person who had never used his business put a 2 star rating a few months ago. He looked at the profile and it showed many ratings posted in quick succession. He reported the review and the profile but it was not removed. He eventually talked to a Google Customer Service Agent and he said their response was this, “The AI is not removing the profile or the review because she did not break a rule.”

He asked me, “How did she not break a rule by rating a business she never used?”

I answered, “I believe the rep meant that from known data there is nothing for the AI to act on because the person could have had a bad experience and posted under another name and you could be a business owner who just wants to get a negative review removed, there is no data to clarify this.” In this instance for the Algorithmic AI “Right” means inaction due to lack of data showing cause to enforce a rule. While multiple ratings / reviews in a short time may trigger the spam filter, sometimes it does not.

Actually, I am not concerned about these views, I am interested in the science and learning how to see the whole picture as best I can. :cowboy_hat_face:

We are all related.

Cowboy Z

3 Likes
2

Hello @Cowboy_Z ,

Thanks for sharing your thoughts with us! I just wanted to let you know that I’ve just recovered your post from the spam folder. Our automatic filters run 24/7 to protect the community from unwanted content and they can be too strict at times, thanks for your patience.

If you want, you can read more about why this might have happened: Why was my Connect post marked as spam?

2 Likes
3

Hi, @Cowboy_Z

Thanks for this interesting post.

My feeling in this case (Google Maps) is that we are just the recipient of the activities of an AI that is trained through other channels.

So in my opinion, if you wants to know how the Machine Learning system works, you will have to search in other communities, for example, as I already told you, in Google Crowdsource, or in Google Translate.

Translate is a great example of how the capacity of understanding a text is improved, especially in the most common language.

Same for images. Have you tried to check how your photos are classified (categorised) in Google photos, and how easy is now to search in there by subjects?

Of course in Google all programs are interconnected, so we benefit of the global improvement of the AI.

The next step is to confirm or deny the choice of the AI, for improving the quality of the response.

The human presence is necessary in every step (we feed the AI, so we are the teachers) but if you really want to explore this aspect, you will have to search in other sides. I am sure @JaneBurunina can go more in depth with the subject

1 Like
4

Hi @Cowboy_Z

I’m trying to understand the reasoning behind your post.

Are you basically critical towards the use of AI to handle all the edits from Maps users? So you want Google to be more aware of some of the issues with the far from perfect automated verifications?

I would like to offer a different perspective. I’m pretty impressed with how the AI spam is working. But I have also spend many hours systematically evaluating the feedback the AI is gives on my edits. I make a huge lot of edits so it became kind of a sport to get to know the AI and predict the outcome of each edit: approved, pending or rejected (could not verify). So while you might think I’m just out to please the AI, I can inform you that lately my immediate approval rate is well above 90 percent.

This is for sure a 2-way process where I limit my edits to those I expect/know will be approves and the system trusting my different kinds of edits more and more over time.

So I find the system super adaptive and very good at blocking spammers and letting my edits get approved right away.

Cheers

Morten

1 Like
5

I am adding some interesting reading, @Cowboy_Z , just popped up on my email:

What’s a neural network?

It can be an interesting starting point

1 Like
6

Thank you for this input @MortenCopenhagen

That is a great description of your process of trust building with the filter! Nicely written.

I am learning in many areas so I don’t feel yet that I have enough experience of the actual activity of the algorithmic AI to be skeptical. Also, I am not a skeptical type of person. I enjoy receiving many people’s views from their experience to help me understand what I see.

My question is evoking shared insight and is not skeptical.

What I am saying in this post is that metrics revealing the performance of the AI would help us understand more what it is doing and why. This is part of the natural fun of working with Google’s systems to help other people make informed decisions while solving as many problems as possible along the way.

We are all related.

Cowboy Z

1 Like
7

Lee Sedol experienced many things through the 5 games of Go with AlphaGo. In game 4, the one game where Sedol triumphed, much attention has flowed to move 78 where a reversal move turned the game that AlphaGo had been controlling. After this move, AlphaGo seemed impacted in the neural networks in the interface between the three aspects of its depth.

In the article you shared @ErmesT , Maithra is quoted by the author in explaining how a neural network can have unexpected gaps. In game 4, the Google team that handles AlphaGo became quite worried over how the AI was responding to Sedol’s move. Many of the machine’s calculations predicting the game action were disrupted and it’s way of proceeding in decisions showed some offbeat, whimsical seeming and inexplicable choices. As Maithra also mentions the relationship between the “neurons” in AI are different than in humans due to the mathematical context. When AlphaGo reacted with what seemed to be some confusion, the team’s concerns connected to the discovery of some specific weak areas in AlphaGo’s processing of move decisions from prior to the tournament. while I only understand bits and pieces, it is possible to see the effects of this kind of processing confusion as the layers of the AI neural network react to stimulus like occurred in Sedol’s move 78 in that historic competition.

To be just a bit more detail focused: the three modes that AlphaGo interlace, bringing together very different math processes such as projection of potential events and comparison of prior scenarios. When the intelligence is leaning, or having momentum in a computing pathway and then is disrupted, there can be some complex shifting and re-shifting like a small boat in a lake bouncing in waves. This disruption and re-shifting is reported to produce unexpected actions and reveals integrity or gaps in aspects of learning.

It was very interesting to many computer scientists to see how AlphaGo recovered in game 5 and demonstrated new competent style elements. Sedol himself was surprised at the creativity that AlphaGo accomplished, and creativity is a strong-suit of Sedol’s famous playing technique.

To tie this back into our post thread, and this may interest you @PaulPavlinovich - especially because unexpected reactions and mysterious inter-linkages of interpretations are known to happen in AI neural networks, the metrics used to grade learning are very important and enable progressively relevant feeding - training for continuous improvement. This is why I am asking for input on this topic of this post. @MortenCopenhagen

As Local Guides reporting data that breaches rules there is definitely a large mystery factor as to what responses will be made. It is good that we study not only how the filter handles our review posts, but also how the AI responds to our reports and the effect these responses have on people and businesses.

1 Like
8

Sorry for misinterpreting your motives, @Cowboy_Z .

Google has the biggest group of AI researchers. Many of them working out of London. The company is named DeepMind. You can read about them here: https://deepmind.com/about

They have hired many if not all the best researchers who initially were free to continue their academic lines of university work.

Their work is super relevant and has an enormous commercial potential for Google and “the world”.

Measuring accuracy is pretty straight forward when you have access to all the data Google own.

Regarding the quality of the spam filter we are concerned about I doubt this is part of the cutting edge research. It does not need to be perfect as long as Google Maps is getting the millions of edits they get daily. Fending off spam is way more important than keeping a few frustrated local guides happy.

Google is a business and makes decisions based on how they can earn money, so don’t be surprised that refining the filter beyond what is nessesary will come later rather than sooner.

These are my thoughts and not based on any insider insights.

I think the AI spam filter can be improved A LOT.

Basically the metrics you are interested in is simply the percentage of correct verifications.

The system is continuously getting better and better based on many inputs: including our actions as local guides.

Cheers

Morten

2 Likes
9

It is an interesting topic @Cowboy_Z there are no public metrics that I’m aware of. This sort of data and the model itself would be part of what I would describe as “secret sauce” so in the same way that no-one really knows what’s in the sauce of that burger chain although there are some very close approximations available no-one is going to know such an item of Google’s intellectual property. Corporate entities are like onions. You get access to the layer of information you need to do your job and that is it. Being on the outside people like us can peer in through the windows and postulate on how something might or might not work and could indeed even construct a pretty good approximation using publicly available tools like TensorFlow based on our observations and training the model with real world data but it would be just that, an approximation.

Paul

1 Like