Cybersecurity and AI – Why do we trust people?

One of these challenges is trying to understand just what the AI is really doing. Traditionally, when we create an application – say, a webshop – we write the code which can handle all the use cases: taking payments, displaying a list of products, storing the shipping information in the database and so forth. AI-based approaches are a little different. Instead of writing the code explicitly for everything, we feed algorithms with data and have them figure out what the best approach for a given problem might be. Today, it doesn't look nearly as fancy as Hollywood would like you to believe. In future though, who knows.

With the webshop case, we know exactly how the shop works. We can pick up the code, read through it and audit it down to the last line until we are happy. Certainly, with complicated enough applications, this might not be a feasible approach due to how much time it takes. Still, in theory, at least we could do it and learn exactly how the application was made and designed if we had enough time and skilled people. With the AI approach, however, things get more interesting. Even today, it’s really hard to take the model that was created as a result of analyzing all the data and explain what exactly it does. It seems to work, and it’s giving us results that make sense, but we can’t always tell what exactly is going on.

Code made by humans has a trait that is rather obvious when you think about it. Because we created it, there has to be some sort of train of thought that makes sense to us. The programmer didn’t just throw things into a blender and get the final product that way. No, they figured out how a webshop should work, what sort of features it should have, how those features play together and what kind of problems we specifically want to avoid. It almost forms a story that one could tell to someone and have it understood with no need to know anything about programming. In fact, programming today is often based on what we call "user stories". These are simple human-readable statements (e.g. "The web shop should accept payments with Visa cards") that come directly from people who often have no idea how the implementation will look like. This ensures that there is a method to the madness of creating the code.

In the future, as things get more and more complicated, it’s hard to say for how long we still understand how the AI models work and whether we can verify or audit them. Theoretically, once a human can no longer follow the logic or it gets too complicated, we may have a few options beyond just relying on that the algorithm is doing its thing because the mechanism that created it is solid. So far, AI hasn't been great at explaining its own train of thought. Certainly, it doesn't give us user stories that would make any sense to humans.

We already have plenty of examples where AI goes wrong. Sometimes it’s really obvious, sometimes not. Well-known examples include situations where an AI starts to discriminate or becomes really rude to people. Scariest situations are those where nobody seems to be able to get the AI to start behaving and eventually they just pull the plug. While some of these scenarios can be quite amusing, there is a more sinister side to all of it. What if the AI in question is responsible or involved in keeping something safe or secure? How do we know that the security of the system isn’t compromised just because the AI is having the digital equivalent of a bad day?