The Hidden Cost of Weak Security: Agencies Best Practices

Music

Jessica: Hello everyone. Welcome to another episode of the Greyd Conversations podcast. I’m Jessica Lyschik and a senior developer at Greyd, and today I’m joined not only by one, but two wonderful guests. We have here Wes Tatters from RapydCloud and Robert Abela from Melapress, and welcome, guys. Welcome.

Wes: Pleasure to be here.

Robert: It’s nice to be here. Thank you for the invitation.
Jessica: Awesome. So our topic today will be all about security. And, I think both of you are very into, security, both on individual, paths. So just, quickly introduce yourself, to the audience so we know a bit more about you. Wes, do you want to start?

Wes: Certainly. I’m Wes Tatters. I’m the, chief technology officer at RapydCloud.
We have a very strong focus on security in our platform. We believe that security should be our host responsibility. And it should be a first class responsibility in the hosting. It shouldn’t be an afterthought. It should be something that, you build your stack from. You work with customers from. There shouldn’t be a scenario where a customer goes my site’s been hacked. Because it should be the host that’s watching and monitoring for them from the start.

Jessica: Wonderful. Robert.

Robert: Hi. I’m Robert. I’m the CEO and Founder of Melapress. We develop a number of security plugins, not the traditional security firewall plugins, but we have an activity log plugin, atrophy plugin, and logging the security plugin. Yeah, I’ve been involved in security for a few years. I’ve been working in the industry for 24 years. I started working with, as a systems engineer for a number of security software startups, and that’s basically where I’ve ranked most of my security by being so, systems engineer. We’ve been hacked a couple of times in the early days. And then, of course, you start learning. And nowadays. Yeah. 13 years ago, I’ve, I founded Melapress. And now we were developing the plugins.

Jessica: Awesome. Okay, so, yeah, we’re having security, both from two different perspectives, one from our hosts and one from a, plugin perspective. And, I think maybe we should talk about a little bit as, like, diving into this topic that, while you focus on certain areas. I think security is not just specifically one, area that or one responsibility for one person or one level of the tech stack. But it’s a shared responsibility of all parties involved.

Robert: Yes, it is indeed.I would say everyone involved in the website, you start from from the web host, the developer, the persons taking care of the website, managing the website, but also the users, like if you have a team marketing people, for example, even though they are not technical, they have – or you have, if you are the company owner- the responsibility to make sure they keep their software up to date, their laptops up to date, and they follow some security best practices.
So yes, it is. It is actually it encompasses everyone who somehow has anything to do with the website, you know, or even if someone just post on websites, literally everyone in their creation has to do something with security.

Wes: I think that’s an important thing that people to understand. We spent a lot of time, working in, the SOC 2, HIPAA and ISO 27001 spaces, which are the, the, the technical platform requirements for, for security. Basically in, in Europe, and, and in the United States. And one of the key things that comes out every time you look at security is that every single part of the stack, every single person involved in the stack has a responsibility, whether it be the, the end user who has a responsibility for ensuring their passwords are safe, whether it be the website owner who has responsibility for ensuring that plugins are up to date.
Whether it be a host, working to ensure that the malware stack, or their firewalls, are all up to date and consistent and it goes beyond that traditional understanding of, oh, will we install a WordPress plugin that does that, does our security? It goes into those things that, relate to access and control. The first guys have got an amazing plugin activity log, you know, that we, that we use internally as well. We, we don’t just provide it to our customers, we use it inside our, our own platforms and dashboards because it’s a great way to know who logged in to know when something got changed in the platform.
Because security is not just about providing access. It’s about understanding the access that you have.

Jessica: Yeah. And that’s like everyone can make mistakes, right? So we’ve had that with clients, like saying, hey, something broke here. What happened? And actually we also use the audit log plugin and we could pinpoint the to “Hey, you actually did that”. It’s not just something bad happened or someone hacked the site. I mean, it’s such as a blame game here, but, like, it’s easy to make it visible to people like, okay, this action has this result essentially. And as I said, it’s not just about like a bad password or something, but like human mistakes can happen like at any level as well. And I mean a good way to, to like be able to showcase because sometimes it’s often hidden in, okay, this has happened. And now what? How did it get here? How did we get that?

Wes: Secondly, we monitor things like active logins. We wanted to see things like failed logins, but we also keep track of when planes are updated, who updated those plugins. You know, sort of and you’ll get scenarios where why has that particular user got 35 failed login attempts today. And then then you’ll drill back and go, hang on a minute. That’s not their IP address. Yeah. Someone who’s found the username and is now trying to hack that username. So they’re actually trying to and they might be trying that through social engineering or some other tool that isn’t isn’t what you expected. So they might be trying to come in sideways through a user account that that might have an administrative emissions. And it’s important from that the, the, the tech stack and from the, the platform stack that you understand that and keep aware of those things and when they’re happening.

Robert: Yes. And it’s also very important. These are all important points. But as Jessica said, this is not just about blame game or not trusting each other. And that’s why I say that’s where user comes in, because all of us, even the most experienced, make mistakes.
So it’s very important. I always say when I speak to people, like if you are designing some security policies, assume that everyone is non-technical because you can do the best, the best, most check policies are that everyone has to adhere to basically. So this is because all of us make mistakes, even the most experienced. And some mistakes actually can lead to a lot of security issues.

Jessica: And yeah, what other, like common causes are there that, like make websites break or getting hacked? What is it, in your experience, the most common to have been seeing

Robert: based on reports we’ve seen earlier this year from some WordPress, security service providers like Patchstack and Security are outdated software, just the most common. And the secondary user issues now user issues, you can break it down and now many other things password sharing credentials, hygiene, maybe not using the 2FA. There are a lot of, of of issues. But clearly because even updates you should when it’s outdated software, it’s it was a mistake because they didn’t keep the software up to date. So the user usually all problems unfortunately revolve around users, what users are doing, if they are maintaining the website or how they are accessing the websites, or what they’re doing to store their credentials and their security best practices. So the user is really like in this equation is one of the, weakest links. Unfortunately.

Wes: sometimes also, unfortunate because of the nature of WordPress site owners. It’s about timing. And what I mean by that is that, a lot of WordPress sites, even business sites, may not have a full time system administrator. And what that sometimes means is that they don’t do plugin updates every day. They might they might have a schedule and they might have really good best practices where on a win on a Saturday morning, I this is my B job. On a Saturday morning I get up at 9 a.m. and I go through my plugins, I do a backup, I create a staging site, I test all the plugin updates and I release them. And so they’ve got a workflow. And that’s the nature of the business. They might the 9 to 5 week days, they’re working somewhere else or they’re working on, you know, their main business. But some of the biggest malware attacks in the WordPress space over the last years have been caused by that scenario. Unfortunately, one of the best known ones, the, was, it was, you know, a site builder, very well known site builder. And a vulnerability was identified. It was patched by the plugin. Oh, sorry. It’s a theme and plugin. It was patched for the theme and plugin and announced to the WordPress space on a Tuesday morning. Unfortunately, the messaging didn’t say the world is ending if you don’t install this now. It said it was a security update and you have a recommended security update. You need to update it. It’s a very complex plugin. You can’t just reinstall it. It could break your entire site. It could change your layout. And so these things have to be regression tested. So people have gone: Great, Tuesday I’ve got the message Saturday I’m gonna fix it. Friday morning I, I coordinate a malware attack, took out tens of thousands of WordPress sites in minutes. Now, it wasn’t anyone’s fault. Yeah, it was a set of realities in the WordPress base. And so the are the only customers that were protected, ironically, were customers using, vulnerability tools. So what that meant is that, unfortunately, WordPress, you know, word things, tight plugins, security, all of those sort of plugins, they deal with malware. But this is just a plugin that’s got a battle on a calculator. And if you knew what you were doing, you could create a, administrative user just by making a call to that plugins API request. And two minutes later, you had to control this site. And lots of customers that morning wake up discovering that the sites were pointing at a online gambling casino. So you type in your URL and you’re in an online gambling casino in Central Europe. So that was a set of situational circumstances. Patchstack, which was just mentioned before, was the plugin that, did protect people because what Patchstack had been doing was even though the customer hadn’t yet updated this thing or updated their plugin, it was stopping the vulnerability. So it’s sometimes about the the tools that you choose to utilize, in your stack as well.

Robert: Yes. And to adapt to what was saying, which is like, oh, look at our classic. We have a wide range of customers. Even if you go to a restaurant, you see a wide range of people, and some people work for an agency, some people work for a big government organization and some people, as opposed to saying maybe have, a flower shop during the weekend in the in the weekend. They like to maintain their website, which is like a common, scenario. But it’s also a problem because unfortunately, the last few years, when a vulnerability is disclosed up until ten years ago, you would have waited weeks until you start seeing it and exploit it. just 3 or 4 years ago. We’re looking at this. So, an advisory is published within days. This vulnerability is being published in the daylight this year we’re talking about hours. So as soon as an advisory is out, within hours, there are people, especially thanks to AI. There are people who can immediately, spin up some scripts and start scanning for this vulnerability right away within hours. So yeah, that’s why it’s also a problem as Wes is saying, because if you have a business and you’re at your flower shop during the day, you’re not going to update your product.

Wes: Yeah. And part of it’s about the messaging from the plugin developers. And so they can’t go don’t don’t do anything. The world is ending. You must install this now, because that’s the best way to tell the world that there’s a vulnerability. So they have to be careful about what they say. Yeah, they have to be. So it’s it’s yeah, it’s a bit of a balancing act.

Jessica: True. So, when it comes to agencies, I think this is probably a special case because even within agencies, I’ve worked at different agencies before. You know, you mentioned, like, okay, there’s, people doing as, a website as a hobby business or a side business and then there’s like, governmental, websites that, need to, require immediate attention. But often like this. You’re in an agency, you have clients. Maybe not a government, but maybe have an online shop or something that is maybe urgent and maybe websites that are not so urgent. And I think what I see often is like, people are trying their best, of course, but like getting lost into one client requires this, the other one requires that. And it’s a real mess about, how security is handled. So what do you think is the best way how you can approach, such a situation like an within an agency?

Robert: Unfortunately, yeah. With agencies. The question is scalability. Like, if you have 100 customers to 101,000 customers, it’s a big problem. I think, the the biggest advantage, the biggest tool you can use, maybe start with auto updates. However, as as Wes said, it depends on which part it is because if it’s a page builder where a lot of it depends on it’s it’s a bit of a problem. However, definitely by enabling auto updates for some particular plugins, especially the minor updates, you’re pretty much safe. So if you build, if you build a system, a framework where you have where you’re always using a certain number of plugins, where you build a techstack with a certain number of plugins, you have a certain test environments, and with experience within a few, within a few months or years, you kind of like, expect and know what to expect when there are updates. So you can say, okay, these plugins can be automatically updated these plugins as soon as they are released. So we can we can run these tests right away. By the way, nowadays, especially with AI, it’s much easier to run some tests automatically, so hopefully that is helping a lot to move faster. The biggest problem again to to add after what was said is yeah, up until a few years ago, it was possible to run a website as a hobbyist kind of thing. Is it like, oh, maybe you forgot to update a plugin or something, or maybe you use some easy passwords or you forgot something, some some of old account is getting because of it. It’s getting harder and harder and harder to maintain a website. It’s not the user’s fault, but it’s getting harder because because of all the tools we have nowadays, even with AI, yeah, we are moving much faster. For example, even if the vendor, if there is a security issue and the advisory is published, even if the vendor doesn’t specify what the issue is, if we are talking about free plugins which are hosted on replicas. Org a lot of people now, they index, they index that, have a local copy on the computer, on their servers. And as soon as there’s an update they do a diff and they know right away within minutes which plugins were updated, which lines of code were updated and if it was a security issue. So it’s it’s almost like the faster the tech guys move, the faster to move. So so it’s it’s becoming much more difficult. Yeah. If you have a bakery for example, don’t you want to maintain your website. But during the day of course you are working at the bakery and maybe in the evening or as we said, you can do on the weekends updates on plugins. This is becoming more and more difficult. And that’s why most of us need to rely on some sort of professional help or agency where they have like a solid tech stack. They have experience with maintaining websites, updating plugins, maybe having some sort of test frameworks in place so they can keep your website up to date as soon as possible. Sometimes, not always easy because as I said, if it’s basically there’s going to be a problem. But for the majority at least, you can move as fast as possible.

Wes: I think, proactive tools, like tools like Patchstack are excellent. I, I don’t tend to just recommend plugins. But I think, Patchstack in the WordPress ecosystem is a very unique tool. There is only IT and WordPress or automatic. So, WP scan is a scanner only. It’s not a proactive tool. Patchstack is the only tool in the space really that has vulnerability management in terms of the fact that it’s got its own bug bounty. It means that people are submitting bugs to them. It’s got a – having been also not just up a host, but also a plugin, you know, I’m a member of a plugin development team that has had vulnerabilities reported to it. One of the challenges is that, and this is a unique challenge for agencies. So most agencies are building their own code as well. They’re creating their own custom plugins. They’re they’re creating tools for their customers. And I think that adds a unique challenge for the agency. See lots of WordPress sites just downloading plugins from WordPress.org. WordPress.org’s plugin team are running scans. They’re running vulnerability checks, they’re running malware checks. They’re using tools like WP scan to ensure that the plugins in the repository are as, you know, safe as possible. But I think in the the agency space there is a unique challenge because very rarely are there plugins scanned, very rarely are there plugins being validated. And I think in those cases, the there’s a, there’s a kind of a partnership with platforms like Patchstack that people don’t actually know about. Patchstack will also work with plugin developers to actually scan their plugins, scan the installed code, and identify vulnerabilities. So I think for agencies tthats… I’ve this is at levels of scale for an agency. If it’s an agency with, you know, ten of 50 customers, it’s probably more of a challenge. But larger agencies in that thousand customer plus area that especially are doing their own regular custom development, I think also have a responsibility to ensure that and and look like let’s be honest, AI is a great tool and we all use it. I was talking with one of the the WordPress plugin, assessment team about six months ago at a WordCamp. And we’ve known each other for a number of years, and, they were saying that they’re getting a lot of plugins that have obviously had AI assisted code. The scary thing was those plugins had malware in them.
Now AI scans the internet, reads all that. You know, it’s read all the WordPress plugins, and it’s using that code as its code base to help it create new WordPress plugins. The challenge is that the the, the code that was written had vulnerabilities in it now. So it’s everyone’s responsibility. So, it’s about doesn’t matter who it is as being a developer or the end user or, or the agency, taking taking an opinion that it unfortunately and it’s not, not being a doomsayer, but unfortunately taking the premium that there’s a good chance there’s something wrong. So make sure you’re doing scans, you make sure you’re doing vulnerability checks. Even to the point, I mean, doing penetration tests and I mean, and again, this is levels of scale. It costs money to do a penetration test. But, you know, sort of if you don’t know what you don’t know, then as an agency, that’s a challenge because your customers are expecting you to know what you don’t know.

Jessica: Yeah. The the customers are trusting the agency to like, yeah, not, make it worse and, than it is. So, I think this is also like, even for what you said, that’s like for, for big agencies. But I think even in small agencies there is custom code somewhere because a client needs something very special. And it’s, there’s just no plugin that covers it.Despite this, so many plugins …

Wes: and sadly, it’s so simple. Yeah, it could be a really well-meaning developer in a company that went, oh, we need a new hook. And they put it in their code. It sits there dormant till someone stumbles across it and discovers that if they call that particular line of code, it flips a switch that changes the user from a, end user to administrator. Now, the plugin developer may have done that for a very specific reason. So it might be something like a view as tool, view as administrator, or view as a current user. And the code worked and it did exactly what it was supposed to do. Unfortunately, if someone could find it and discover that it was there and discover that unfortunately, that particular API call didn’t actually have a, logging requirement, so it was publicly available and authenticated.And yeah, and we’re talking about something, you’d go, oh, surely that can’t happen. But that’s exactly the vulnerability that occurred in that, same build that, that we’re talking about. It was literally an API call. It didn’t have authentication on it. And anyone could call it. So yeah. That’s crazy. Yep. There’s there’s no simple solution. But, definitely tools like Patchstack. Definitely tools like WP scan. Don’t just rely on, like malware tools don’t just rely on vulnerability tools. And I would also suggest that agencies, look at partnering more closely with those type of vulnerability type plugin developers, look at developing close relationships with them because they are proactive people, like all of us. I see, very proactive about, helping people in the WordPress space.
Similarly, developers, sorry, like malware security at server level type developers, like, monarchs, in particular are very proactive in working with agencies and working with hosts, to help them, you know, I mean, we use Monarchs. And I’ll be honest, it’s a full disclosure thing, how the Monarx team sits inside every one of our customer’s websites in real time, monitoring what’s going on. It’s it’s not a plugin. It doesn’t slow things down the plug. The customer can uninstall it. They can’t deactivate it, but they sit there monitoring in real time. And what that monitoring in real time does is it becomes like a honeypot. If they if they see us at a specific strange activity happening, there’s a, there’s a regular activity happening on WordPress sites today that’s opening a folder or trying to execute a folder in a temp directory or something like that. They very quickly could dogfood, and, and honeypot those things so they can actually start seeing them even before vulnerability has been published, even before malware exists. In the last 12 months, I’d say in our platform, we’ve, we’ve seen maybe 15 or 20 zero day attacks that no one knew about. They didn’t, you know, and and then they something suspicious starts to happen
And that’s where you’ve got to rely on your, your, your security partners to help you. And those suspicious things can get escalated very quickly. In Monarch’s case, about 15 minutes. And they can start vulnerability patching their entire customer base or our entire customer base as well. When they see something suspicious happening. So I think it’s important for our agencies, especially the bigger agencies, to develop relationships with those various, malware and security and vulnerability providers as well.

Robert: Yeah. And just thought up also, what was it saying? Because you mentioned a number of tools, it’s important to point out there is no one tool that fits all of those and and does everything. That’s why it’s important security about, processes, best practices and a suite of tools. And I just want to… so once you build your own tech stack of security tools, best practices in place and stuff, it makes life much easier.

Wes: Yeah. And develop relationships with them. Have that really close sliced. Yes.

Jessica: So yeah I think with with security it’s like for the very average WordPress person, it’s just like, oh hey, let’s install this security plugin. And then everything is fixed. It’s like with caching, let’s install that caching plugin and then everything’s fixed and our website is yes or yeah. My website is fast but not really fast. So for some reason I don’t know why. And it’s the same actually with caching and with security. It’s like there are so many different layers that are responsible for you, like having a fast site or having a secure site. And I think what really not probably everyone knows is like, what are these layers individually? What you can you do at what point? And I think this is where it gets interesting. To learn more about each layer of okay do i need a two factor authentication or should I maybe, how often should I do backups or whatever on whatever level you are thinking about? But it’s not just having backups at some sort of having two factor authentication, but it’s also like having updated plugins, of course. And I think it’s just very much like the thinking of how you handle security is such as one thing fits all, but here’s a list. Here’s all the things that we should take into account and I think this is what people probably, need to hear a bit more my thinking for, though.

Wes: So I think it’s important also to develop language, between agencies and customers, and hosts and customers as well.That helps. Helps refine some of these messages. Very, very recent example, was literally last week. We, were working with a customer who, had been migrating into our platform. And they migrated in and started going, there’s problems with our site. Can’t access this particular page. Was a really good example. And, so they’d gone off and fiddled and this solution in that case was the turn off collapse live proxy, because it was obviously Cloudflare was broken and the site was this site wasn’t a problem. Unfortunately, when you drilled into it, the cleft lip proxy was blocking that page. It it actually detected a vulnerability on it. The person was responsible to turn off the cloudflare proxy, which rendered the vulnerability was opened to the universe. So it is important to develop conversations and dialogs between, you know, the agencies and customers and obviously hosting customers as well that go, let’s stop looking at logging tools. Let’s start looking at activity log. You know, viewers, let’s start looking at, you know, Cloudflare logs. We went into a collapse of logs and I was like, yeah, look, vulnerability on their page every time it was being blocked. Blocked block, block. So having language in conversations that say to customers, if there are 4 or 3 errors, as a classic example, the reason is largely because your firewall or your vulnerability tools, saying there’s something wrong, don’t turn the tool off. Identify what the actual problem is. We’ve seen similar issues with platforms like Patchstack. My site is not working the way it should. Patchstack is broken. I’ve turned it off. My site’s working fine. It’s the same issue. Yeah. No, no, no, no, Patchstack was telling you something was wrong. Or I’ve decided. Wordfences blocking all of my IP addresses. Okay. There’s a really good reason for that. Those IP addresses, have been identified as from a region or territory where those particular IP addresses are being up in everybody’s malware targets. So it is important, I guess, for us to have conversations with customers and relationships where people understand that security is becoming really a prevalent issue and and solving problems by turning off security plugins. Is is respectfully not a great idea.

Robert: Yes. It’s a very good point. We have similar experiences on a different, environment. So people when people start using our plugin deactivated log, they ask us, okay, what should I look for? It’s like, it’s not up to me. That’s one you should you know. This is your website?
You know who logs in at what time? You know, from where they look at, you know what they should be doing and what they should do. So based on the information, then you should be able to keep an eye on the logs. If you have a user, for example, and he’s not allowed to access page A, but the logs showing are showing that, they are accessing page A, it means there is something wrong.
Now if it’s either a security issue or maybe a role misconfiguration, but there’s a security issue there. But me as a vendor, I cannot tell you what you should look for. The you should have a better understanding of what your website is about. So as a website, website owners, even though sometimes it’s easier to say, okay, there’s a good solution that’s just implemented and is done, they need to have a better understanding of their website. What is it doing, what the tools that they are using are doing. And yeah, because by understanding that if there are any problems like this is sake, or if there are, or if you want to learn a bit more like, okay, what are the logs saying? You need to understand the website. I cannot tell you how someone is using the websites. I don’t know, somewhere like far away from my like somewhere in Japan, how they when they log in and what they are using it for it. So I cannot tell you what buttons to find the logs. So we should have a better understanding of how your site is used, who logs in and that’s what time. That’s what they do. And then you can use the logs appropriately. And this is the same, exactly, and a different scenario. That’s what this is saying. Oh my father is borrowing things. So I show that I shut down the firewall. If you don’t understand the tools and what you’re looking for, then of course there’s a solution. Yeah. Remove the plug in. I don’t want to see all these logs shut down the firewall. I want to see this page. But you’re making a mistake. So the users are also the owner of the website. It’s also responsible for its own security. Or at least as Wes was saying, I think the easiest thing is to open a communication channel and to talk a bit more on what you’re looking for, what your concerns are, and you find the professional people who can help you solving this issue.

Wes: I think also, Oh, sorry.

Jessica: Go ahead , Wes, go ahead.

Wes: I was gonna say, I think there’s this one conversation that is really relevant at the moment is AI, and the fact that, you know, AI is, is evil or regarded to a certain extent is a little bit evil in the world in the the vulnerability space because it is now so simple, to create, vulnerabilities, vulnerabilities. Yes. As we were saying earlier, that they’re appearing every hour now and they’re largely by AI, but there is an interesting use of AI. And a number of our customers are using it now, and it’s to actually help them read the logs. Yeah. You can literally and we, we, have some very big customers who they don’t have the bright space, the time space or anything to analyze their own logs, but they can pull the logs now into a very good AI tool.
And there are a number of the math I’m not going to mention them, where they can simply go every day. Hey, here’s my logs. Read them, please. See if there’s anything of issue and within a couple of minutes, AI tools will be able to identify issues in what might be, you know, terabytes of log information, and can actually help customers be proactive, help customers analyze, big complex logs of like, here’s my error logs that I on my access log. Please just look through it, look for it, look for suspicious things. And so it can be used quite actively as a proactive tool by end user WordPress owners that aren’t necessarily technical in any way, shape or form. So it is, potentially proactive use of, of AI, that can help, site owners, site administrators, to do things that they never did in the past because it was just too hard you know, to I, we didn’t have an hour a day to read through every law in the real log or look at every error on the log report.
So for unless you had to because like there was some sort of issue we had to fix, you’re trying to follow the developer. Yeah. So if you could change it to a proactive thing where you go, even if you audit doing it once a week, you might not be a massive site. You could point logs once a week. And, and an AI tool can respond in minutes and say, hey, I’ve identified these ten things that you should look at. Or more interestingly, what I can do is look at things that, over time, I, I’ve identified that every Monday morning between 9 a.m. and 10 a.m., you get a bunch of suspicious requests for a bunch of suspicious IP addresses. And and you, ironically, would probably never even find it yourself.

Robert: It’s funny you were mentioning this was cause I was just experimenting myself just a few weeks ago. We just have some hundred boxes. We’ll just get all of the logs from from the WordPress logs, the access docs from the web server. And it’s, it’s, even without having an AI specialized agent, it just just, you know, just put it into check. Yes, I just does feeding it like, listen, these are the logs from my website for the last month. Yeah. Tell me what you see. And the information starts. Get giving you. And you can start, diving deep, for example, you support something. Okay. Can you please tell me more about what this user. When did he log in? What did he do?You know. So imagine. And if you if you save all this information and every week you are feeding it this information, it starts. Then of course building patterns and then it can catch anomalies. So for example, oh Robert usually logs in between 9 and 12. And from this IP address. However last week I logged in at midnight. But from this IP address.So it’s like and those things yes I can catch just about three. You can spot them right away.

Wes: Yeah. And you do things that you’d never spot yourself.

Robert: No. It’s impossible unless you configure some alerts because there are like some solutions that say okay if someone then from.

Wes: Yeah. But you have to add to what might have been that Robert was on holidays in the Bahamas. But in other cases it’s someone who’s in, you know, a third in Western Europe or in Eastern Europe trying to hack this site except with and so but they’re great tools to and and there’s things that can do things very quickly and simply that and user hosts and also, I guess and also something that agencies can be considering as well, you know, agencies could use those same tools to once a month access the customers logs, pulls them into an AI did a bunch of proactive reports and use those reports to get back to the customers. For multiple reasons, i.e. to obviously, provide them feature service, but in some cases it might be identifying new vulnerabilities. It might be identifying, you know, for the agency, it might be identifying work points, you know, and then new service points, for the, for the agency as well.

Jessica: True. would you I mean, this is a very theoretical question. Would you trust AI saying, hey, can you secure my site or make it more secure? Or would you say just no, don’t don’t don’t do it.

Robert: It’s it really depends. Is it like one thing I’ve not I thought about, I was reading a lot about is first of all, you should, you shouldn’t ask very general questions. You need to give it’s way more context and build blocks by blocks. So if you go for something like which probably most users do, especially if they’re thinking, hi, can you secure my website? I remember speaking to Oliver from Patchstack this year, first of all, how many plugins they’ve seen from people prompting AI: “Hi, can you build a WordPress security plugin for me?” As simple as that. That is too jarring.However, if you if you are technical and you know your website and start building block by block, for example, I’d like to have an accessible to block the support, for example, or to block these type of requests. And you started building slowly then yes, you are using AI as a tool. However, if you go to AI to Jupiter say, hey, please develop a WordPress security plugin for websites.
Yeah, most probably asking for trouble and you are actually was probably introducing vulnerabilities in your websites out there, then closing issues.

Wes: Right. Yeah I think, I think so AI is misunderstood. We’ve reached a point where we think AI is amazing and it has amazing use cases, but, anyone who’s bored have a conversation with ChatGPT and ask it a bunch of questions and then ask it if it’s sure.Are you sure about that? And inevitably the response will come back. Oh, hang on, let me check. Or yeah, you know, let’s not maybe. Right. Let me just go and check some more. And I will tell you that it’s wrong, because the thing to understand is that AI is just using knowledge trees. So you ask a question and it starts building a set of knowledge trees based on information.It has. But that information, is only as deep as the knowledge tree search. A really silly example. A number of years ago, Tim Berners-Lee, who, created the World Wide Web, came to Australia, for a World Wide Web conference. And, I was managing media for it. We got to spend a little bit of time together, and, I was only recently I was doing the presentation.I went, oh, I can’t remember when Tim came to Australia. I want to talk about a something that he was talking about way back then. So I asked ChatGPT, who told me that he’d never come to Australia. Okay. And I was like, I, I’ve got a photo of you, him and I standing at the… you know, he did, but ChatGPT was adamant that he’d never come to Australia. And so, context is important. And I kept on asking it. I said, are you really sure about that? And and we keep digging it and eventually found, on the one of the W3C archive pages in Wayback Machine, the press conference and the actual, you know, was the attendees anyway. And they goes, oh, you know, you’re right.
So asking ChatGPT two to create a, a security plugin for you, is fraught with the same challenges. Oh, are you sure this will protect my site? I’ve looked in the. You know, you’re right. Maybe it might. So I think, as I said, as Robert said, building blocks, using it for the things that it’s really good at.
Help me to write out. Help me to write this access rule or, you know, sort of help me to scan, asking it to scan your site code. Yes, it can come. It’s good at it could scan all the source code in your, in your platform, asking it to scan the information in, you know, do an SQL dump and, and and ask ChatGPT to scan your database. Yes. It can do that and actually produces some scary results sometimes. So there are definitely things that can be done.

Robert: Yeah, it’s much better definitely at analyzing things as opposed to saying if you feed it like, hey, I have these plugins, these versions, this is my database, then yes, it can give you more context like, oh, I spoke like this, an older version of the plugin or in the database there might be some settings which are no longer use or could be as a use issue, but going the other way around for it, like develop a security plugin for me, then it’s it’s going to be a problem.

Jessica: Yeah, yeah. So what do you think is like. So if you’re an agency and, like hearing this, podcast right now and then you’re probably thinking, how should I really approach this? And also, okay, I maybe I’m using a plugin or maybe I’m doing some sort of backups. What kind of, suggestions would you give a person, that runs an agency to, like, make sure that, they are actually, like, not making themselves more work, but instead having it a bit more streamlined when it comes to security. The whole process,

Wes: backups. An interesting question. And this is an agency responsibility that I’ve seen and and, and, and developers who support, websites, someone in your team should be responsible for ensuring that all of your customer backups are real. Just because the backup program said it’s done a backup doesn’t mean the backups are real. They may. That right? Yeah. Yeah, yeah, yeah. As sites, we’d be back as we run the every day. And if something’s going wrong as they try to restore. And what they discovered is that something in the database was corrupted or something somewhere was corrupted. And in fact, the backups suddenly got missing half the data cause it was continually failing. So there are responsibilities, to, in terms of security and in terms of, best practice in terms of things like database security we ensure that, yeah, the minimum at some stage, at least once a month, someone should be making sure that every customer’s backup actually has a backup in it. And it’s restorable, in some way, shape or form now and again these days, I can help with those sort of things.It’s not specific, but that is a best practice that, is forgotten. We just assume it. I saw I saw a conversation in, you know, in, was WordPress group only a couple of weeks ago from someone going, I, I’m moving. I’m moving hosting companies. And I did a backup. And, and I’ve moved to a new host and I’ve restored it, but all these things seem to be missing.It’s like you probably should have made sure your backup was good before you deleted the old host. And they were sort of like What do I do? And it’s like, well, it was a very well known host that was coming from. And I would suggest they go back to the host who actually still had legacy backup. So from those hosts will hold, you know, security backup or an image for, for a month. And in that case, we understand that they managed to get the information back for them. But, you know, just making the assumption that the backup is good. I think is not a good look, you know?

Robert: I think as an agency as well, to add on top of all these things, as if you start, if you look at the next 24 hours, what can you do to improve situation? If you start with some basic best practices or the good step like seek finding a way how to keep the plugins up to date? By the end, while in the meantime protect them. Use a solution something like Parsec, for example. And so while you do some testing, at least you are already covered. And then of course, if you take care of some basic best practices, authentication hygiene, for example. Yeah, strong passwords using 2FA and keeping software up to date. You are already a step ahead of almost all the type of automated attacks. So by by taking care of the best practices, the basic ones. Then of course, once you nail those okay, software is all the time up to date, or at least when it is because we cannot do it because we need to do some testing. There’s something protecting it, and you have to have faith and all the users have the correct rules. You’ve covered the basics. And then of course you start building on top of it because of course you can go. There are so many layers where you can go for example, you can there are CM solutions where you can aggregate, for example, all the logs of all your customers in one place and have some sort of analysis there, but those come later. So if you take care of the basics and best practices, up to date software, secure authentication, proper user roles, you are already Like in a very good place in terms of security. Yeah. And, I think one of the biggest problems I’ve seen as well, which is a bit of a sometimes it depends who you speak to. But with agencies, some agencies, even though they are responsible for their clients websites, they allow the clients to be administrators, which which I find so frustrating because then the customer isn’t technical, then they choose something, then. Oh, sorry, my website is broken. If it’s just about trust, but I would personally one of the best practices like the customer maybe can have access to update plugins, maybe change something settings, but they shouldn’t have access to change to plugins or to change some settings. So yeah, all this type of basic stuff, if you start implementing these basic steps, you are of course, this is not just about getting hacked, but also about, the operations keeping the websites and the uptime. Yeah, these these things slowly, slowly will help you build a good stack, a good number of best practices that lead to much less problems, both from the security point of view, but also from the technical point of view analysis.

Jessica: Yeah. So let’s maybe talk about what to do when things go wrong because it will happen.Even if you have like everything up to date, it may happen, may be happening just as Wes described it earlier with it’s so far so, with the help of AI vulnerabilities can be, made public and then be abused basically to, hack sites. What would you do or what are your suggestions towards. Okay, something has happened.The site is broken. What do you do?

Robert: Basically what I would do, I would just take a snapshot of the website as it is and restore it to offline, somewhere offline and start analyzing what’s what’s happening. Maybe, some account has been created or some old plugin and start building from there based on the by analyzing the logs. By the way, when I say logs then just do a perspective for the logs. What you have, the weapons activity logs, you have the access locks of the observer. You have the error logs, you have them. There are a lot of logs which you should analyze by doing some forensic work and trying to go step by step and trace back what happened. Then you should be able to say, okay, this is what happened. I don’t know. We had an old plugin. It was hacked. There was it had that vulnerability, it was exploited and they were hacked. And then you can start slowly, slowly start to think of changes. You can also, if you know when the website got hacked, you can also restart a backup that you know is a good backup because there are cases even it happens even to big businesses where it took them six months to realize they were hacked. So if you don’t have backups which are older than six months, yeah, you’re you going to restore a version of the website that is already hacked. So so yeah, but it’s important to analyze the logs, try to trace back what happened and start reversing those changes. If a user on the scooters did it, that’s one. If scan your code for malware, try to reinstall all the plugins right from the from the source, not from from your backups. For example, if you had some plugins ABCD installed, download them again, install those and start slowly, rebuilding the websites. I got to replace all the, or possibly tampered codes or any possibly newly created users.

Wes: And the result of those I think part of the challenge, as well, is that.
We talked a lot about backups, obviously. But in some cases, restoring and backup, does as much damage as the vulnerability. And a good example that is, say, a WooCommerce site that’s, you know, got a lot of traffic. They can’t just restore a backup from last Monday. No, they’ve just lost all their transactions. I just lost all their data. So I think in those cases, we need to start externally. We need to start looking at, at how the vulnerabilities get in, and, and looking at, in terms of best practice things like, Cloudflare has got a couple of bad press events in the last two weeks. But tools like Cloudflare, that can sit entirely outside your platform now, Cloudflare is not the only one. There are some other tools out there as well. But, you know, banning it, sitting is also similar. But tools that say the first point of entry to my site isn’t on my server. The first point of entry is a Cloudflare gateway or or banning it gateway or some other sort of, you know, sort of, proxied, entry point. What that allows us to do is have a set of logs that are external, and we can actually look at those logs and say, there, yeah, there are the entry points. And first thing to do is close those, because if you don’t close those entry points, you can be banging the way inside your site trying to fix the problem while they’re still coming in through the gateway. So starting outside, securing that firewall, securing that point of entry. And once you understand that, then you can start internally. You know, doing builds and repairs. Then you can start looking at, and those access logs can also help you understand what happened. And potentially in a lot of cases, identify where the vulnerabilities, likely are. Because if you can see that, this particular call is being made, and that was the thing that’s triggered the vulnerability, then it helps you to build an understanding. I will also say this and some there are some bad press. Tools. And there is a good press tools on this if you’re, if you’re a live business like, WooCommerce site, I strongly suggest that if you have these sort of things that you have a serious conversation with a vulnerability company who can help you make good, it will cost you money. But they have abilities to look at things that most customers don’t even think about. Really classic example from back to years ago. I a vulnerability got into the WordPress ecosystem, through a different builder. And one of the things that it did was they actually hid a really innocuous looking line of code in the database, and it was basically attached to a post record.So it did a post, like a page builder post. Really innocuous line of code. It didn’t look like a vulnerability as such, but when you use it, it was a vulnerability. It had never been identified, but it sat there dormant for 12 months, and then one day woke up. Okay. Now people had the vulnerability, the malware plugins. A vulnerability scanner or a company that specializes in that took one look and went, has anyone seen this before? Because they had tools like AI, tools that could analyze, not. So there’s a danger with vulnerability tools, a lot of vulnerability tools use what are called signatures. So they discover something. They create a signature, which is what it looks like.And then they scan your site looking for that signature, which is, you know, might be this, this 50 characters in a sequence. But if the signatures, evolving if the, the line of code that’s inject it was clever. So what do basically did was it it what it injected was different on every website.

Jessica: Oh yeah. So that’s not like repeatable.

Wes: And you know, it was wasn’t so everywhere. Yeah. What they originally discovered was that the last six characters.Controlled what the rest of it should look like. And it was just every time it was running, it was running totally different. And just the last six characters was a number, and that number told it what the rest of the, the the thing which ended impact that it was releasing. But, When your live site when you’re busy site conversations with, you know, respected, you know, security and Monarx, Patchstack, Wordfence, tools that actually will go out and help you recover a site that has malware in it. I think a good practice for those sort of scenarios, where you just can’t you can’t just restore back up to a month ago. You need some to help you pull your site back together. Recover the data, and, and try to get yourself back to running site. So I think there a different scenarios depending on the sort of site you’re running as well.

Robert: Yeah. And it’s important to point out as was, Wes said, yeah. This might not be something for the average WordPress user. It needs to be quite technical to understand. Even if you managed to find out through the logs what happened to identify which, especially if you have a custom theme and you don’t have the original code, it’s what it will be. Unless you’re a developer. It will be even some developers to be quite difficult to identify exactly what was changed and what happened. So that is why you will need, specialized help that can help. Thankfully, nowadays there’s also AI which might be a bit of help, but still, in most cases, you might need some specialist, some professional help. Exactly.

Jessica: Yeah. I think one of the other things is like clients or even agencies may start panicking like crazy because like, oh my God, the site is down. I have no idea what happened. It’s redirecting somewhere or it’s not showing strange things on the website itself. So I think also communicating is like a big part of, making it not worse, but instead like, okay, calm down, we will identify what’s going on. And then, try to get the site back up in a secure state. Sort of.

Robert: Yeah, definitely. Definitely. The communication is very important viewing such situations. I mean, clients, especially gives you… the less technical they are, the less they understand, the more they will peck. So.

Wes: And I think that’s a part of the challenge. The knee jerk reaction is it’s a bit of a scary thing. My site is down. I’m in the middle of a Black Friday sale. Yeah, yeah, I’ve just spent $15,000 on marketing, and my site’s gone. Sometimes. Yeah, it shouldn’t happen, but sometimes. Hey, if you’re a vulnerability and a malware target, that might be a good time to try and, you know, be malicious as well. So I think knee jerk reactions in themselves is something we need to be careful. Yeah. And one of the things that very few websites don’t have is a best practice recovery policy. For example, in our, business, as a part of our SOC2 and ISO 27001, we actually do, table tests once every quarter where we pull in all of the team, we create a scenario, and we actually have to plan out how we…A lot of the things we’re discussing today, we’ve identified in our table table testing. Now, even for a small site, I think it is, and certainly for agency sites, I think it’s important to actually have some policy, actually have some best practice. What would happen if and at least have one of the classic examples is I’ve seen websites that, my site’s gone down. All right. So have you got the data? Who owns the backups? Where is the backups stored? Does anyone know what the login password is? Yeah. Silly. Silly stuff. Oh, yeah, but the devs, the devs on holidays. Yeah. I think best practices in terms of, you know, what we’re talking about earlier. Start with best practices in, making sure that the passwords are stored on a post-it that, or that. Oh, but also when people are using tools like 1password or tools like that, that, that, that I can’t have corporate access, passwords are stored securely in safe places. You know, if we had, a site that. Yeah, it had gone down, it had a bunch of things and a bunch of the plug ins with license keys now, and you were the license keys were to even try and recover them now, when you were, the logins were. Oh, yeah. Oh, yeah. Someone, someone bought all these, like, someone bought these plug ins a few years ago. And, I don’t know. Or the person who bought them had an email address that had left the business and was no longer there. And they were so starting with some best practices around, you know, even, you know, how we store our data, how we store our passwords, how we store access to those important bits and mirrors nation that at some point in an emergency you might need, and, and we don’t had the luxury of running around trying to find the email or phone number of an employee that longer works with us. But, you know, over two years, but seems to be the only person that has had access to the backup database. Silly things, but, yeah, I think, you know, having some hygiene and some best practice. And, and I think agencies can lead to a certain extent with their customers in that area as well, because their customers certainly obviously don’t have a clue sometimes. And I’ve seen this with like, oh yeah, that my agency installs all of the plugins. Yeah. They probably should have some hygiene around that as well. No.

Jessica: True. All right. So, we’re and we’re coming closer to the end of the episode. And I have one last question I would like to ask you. And that is if you could make every agency owner adapt, adopt just one security practice tomorrow, what would it be? We talked a lot about very different things. But what is the one thing that you would say, start doing this as of tomorrow?

Robert: I have actually I have two because I of course, one of them is it’s kind of like they depend on each other, but, up to date software and using software, definitely the best things you can do, because if you had 2FA but you don’t keep software up to date, you’re still going to get hacked. But if you keep software up to date but you have very weak authentication hygiene, you’re also going to get hacked because those are database issues. So it’s kind of like they work hand-in-hand. There are much more many other things you can do. But if you start with those two, I know you guys caught my sorry, but if you start with those two, definitely you. It’s fine going in the right direction. You and for me

Wes: as an agency, just start to build in just strictly in terms of security staff developing relationships with the companies out there that are expert in this space, with, you know, develop those relationships. You know, we’ve mentioned all over here today and they are really approachable people. Yeah. Develop relationships with them. Look at how they can help, you know, they can work with you, for larger agencies. And I’ll, I’ll try to add two in as well. Strongly consider over time looking at things like SOC2 and ISO 27001 compliance for your agency. They, tasks that can help you and then their numbers and their alphabet letters and those sort of things. But one of the amazing things that they do is help larger agencies, in particular, develop best practice in things like security, by helping you with those tasks, silly tasks like, our role with my employees on a laptop that’s at least got encryption. On its, on its data, or at least has 2FAor password log out, screens, so that we as agencies have some cleanliness, you know, how do we know that? You know, our, our developers laptops aren’t sitting open, you know, on in, you know, in a workspace. You know, we work somewhere around the world, and anyone could sit down and and look at the laptop because it isn’t being turned off when they walk away for two minutes. So tools like SOC2 and ISO 27001, can help agencies develop best practice. Unfortunately, they’re not cheap processes. But, I think for larger agencies, if you got 2000 customers, I think, you know, in that space best practices, something that, agencies should be looking at as well.

Jessica: Cool. Thank you so much, both of you, Wes and Robert, that was a very, very cool episode. I learned a lot. It’s always, cool to, listen to professionals in their field, and. Well, thank you so much for being here. And everyone else. I hope you tune in to the next episode. Bye.

Wes: Bye, Thank you for having us!

Robert: Thanks.

Music