The White Ops Satori Research and Threat Intelligence team released an investigation into ICEBUCKET, the largest Connected TV ad fraud scheme to date. The operation impersonated more than 2 million people worldwide in 30 countries and accounted for 28% of programmatic CTV traffic at its peak. To better service the ad tech industry and those potentially affected by this operation, we held a webinar moderated by White Ops VP of Customer Success, Megan Baker, with Dimitris Theodorakis (Director of Detection), Dr. Mike Moran (Data Scientist, Detection), and Aaron DeVera (Threat Intelligence). Below are some of the big questions they helped answer:
Question: What patterns did you observe that lead to the discovery of ICEBUCKET?
Answer: For obvious reasons, we are not able to reveal all our methods. We don’t want fraudsters to get a hold of our secret sauce, because then we wouldn’t be able to do this type of work. It’s important to note there's still some small proportions of ICEBUCKET traffic out there. We're actively monitoring and stopping this traffic, as it's still flowing through our systems.
The way that White Ops works is that we have a combination of security researchers who pick apart how devices, browsers, and apps work to figure out the differences between what real traffic looks like. Then, we look for discrepancies—we examine traffic that we see to determine if (and if so, how) it’s different from what it’s supposed to look like.
We can also look at discrepancies from other behavioral signals, like where that traffic is coming from and how those devices are being declared/used, then we're able to build up a lot of really good evidence behind what we expect human traffic to actually look like. We combine those aspects of security research and intelligence research and dig through the vast amounts of data we collect from the ad tech ecosystem to figure out what is going on.
Q: Are the owners of CTV devices at risk?
A: This particular operation is a spoofing operation, so none of these devices actually exist. We see a variety of different devices—in terms of how they're declared—that look similar to ones that you might have in your home right now, but nothing is actually running on those devices. Spoofing operations like this are false representation, meaning the traffic is representing itself as something that it's not. In this case, the traffic was representing itself as an SSAI provider delivering content to CTV devices, but in reality, it was just a fraud scheme operated in Virtual Private Servers (VPS) with no real devices involved.
Q: How do fraudsters make money from this?
A: The monetization model is pretty standard. These bad actors sit deep in the supply chain where, from a supply chain perspective, the relationships are more opaque. By hiding themselves deep in the supply chain, they can monetize in a way that is more effective. They can jump through different publishers without having to adhere to standards—like sellers.json—and the lack of standards across the industry makes it much easier for them to successfully monetize their operations.
That was one way that we saw them monetizing, by actually building these fraudulent accounts and the traffic they were monetizing was part of this operation. These are cash out publishers; they are dedicated to selling fraudulent inventory. The vast majority of the inventory that these publishers are offering in openRTB is fraudulent and part of this operation. And that's how most of this inventory was transacted. A smaller portion was ICEBUCKET inventory mixed up with legitimate CTV inventory from direct publishers.
Interestingly, we started seeing traffic that we would consider ICEBUCKET-related mixed up with what we think is organic traffic from legitimate, known publishers. That could indicate one of two things: it could either be the case that the publisher is actually sourcing traffic from a fraud as-a-service model which we haven't seen commonly in CTV spaces. Alternatively, the bad actors are intentionally creating noise in the system to obfuscate their actions. The ICEBUCKET traffic from that legitimate publisher is probably collateral damage in that scenario.
We can’t tell whether it's fraud-as-a-service or noise in the system for the legitimate publishers, but it's a very interesting observation, especially for Connected TV traffic. We've seen that a lot on the web, we've seen it on mobile, but not that much on Connected TV's.
Q: Are certain devices more susceptible to this type of scam than others?
A: Within the report, we give a breakdown of what devices were being spoofed by this operation. While we don't know exactly what was in the fraudsters’ minds for picking these devices, we can make a couple broad generalizations. The largest proportion of the traffic, especially at its peak, was spoofing Roku devices. Roku is well regarded as a secure and high CPM (cost-per-mille) platform with engaged users, so that makes them a target for a lot of these operations. If a fraudster sees that and sees that they can make X-times as much by spoofing Roku as opposed to spoofing something else, then they'll try to spoof Roku.
With SSAI spoofing itself, it's even more straightforward than if you're trying to act like the device itself - you just have to act like a data center server. Interestingly, the ICEBUCKET fraudsters also mimicked other devices that you wouldn't expect to be spoofed at all. Fraudsters are looking into the low volume devices (as a proportion of CTV devices on the whole) as a way to say, "Well, if this device isn't commonly used, it might be less likely for someone on a security research team or a threat intelligence team to break it apart and figure out how the device actually works."
Q: Were the spoofers between SSAI and SSPs pretending to be valid SSAI or between SSP and DSP pretending to be connected to a valid SSAI server?
A: Usually when we consider SSAI spoofing, they're sitting between what they're declaring as edge devices (cell phones, CTVs, game consoles) and the SSPs. They are submitting those ad requests and consuming the video advertisements to reportedly stitch those into the videos themselves. If they sat somewhere else in the ecosystem, we'd likely call this operation something else. Based on the signatures we've seen from both the bid requests and how those ads are being consumed, it sits between the fake edge devices and the SSP's.
Q: Is direct publisher sold inventory less susceptible to SSAI spoofing than when it's bought programmatically? And if so, why?
A: Both models are equally risky when it comes to SSAI spoofing. The big difference has to do with the scalability of the business model. As long as you can scale your operation by having cash out publishers sitting in the supply chain, you don't have to go through the process of working with direct publishers and mixing up your inventory. However, both direct publisher relationships and cash out publishers can be equally impacted from SSAI spoofing.
Q: Which party in the supply chain needs to be more diligent and better vet these SSAI servers?
A: From what we've seen, most of the DSPs were not impacted significantly by this operation, which says a lot about the collective effort we've made over the past few years to secure the ecosystem. The vast majority of inventory was sold through the cash out publishers that were deeper in the supply chain. In the blog, we explored things that we can collectively do as an industry. Adoption of the sellers.json standard is a good start. The vast majority of the cash out publishers were not actually declaring their names in the sellers.json files. We also talked about broader industry efforts that manufacturers and CTV platforms could help drive.
It’s important to note that while this is a SSAI spoofing operation, it is acting like a SSAI provider. It isn't trying to impersonate one particular SSAI provider. SSAI itself is a good method to provide advertisements, especially to devices with low computing power. But there is a lower level of transparency with those SSAI providers. Some of these standards can help provide more clarity into that ecosystem, too.
We don't want to say that an SSAI provider is purely at fault just because they happened to look like them. In the same way, the various devices are the targets or the victims of this operation.
Q: If White Ops was working with a publisher directly, it wouldn't have stopped the ICEBUCKET operation considering the SSAI app is being spoofed at the SSP level?
A: Not necessarily. Working directly with publishers allows us to have a better understanding of what the ground truth of their traffic is. For instance, if we can see the exact list of app IDs as they're updated and what devices they're running on, we can easily tell the difference when we see a similar app ID or the exact same app ID but from a different location in ad tech ecosystem, which would allow us to protect the spoofing of their app even beyond the SSAI spoofing part of it.
Since we know exactly where our publisher is operating, and we see something outside of it, we can tell for sure that it’s spoofing because it's not a part of their inventory. To a lesser extent, we can see where they're trying to do similar copycat stuff. If you are publisher X and your app IDs are declared slightly differently, but it looks about the same or has particular sub-strings within it's app ID that make it appear to be the same as your publisher, we can defend against given the direct relationship with the publisher.
By working directly with publishers, we can significantly improve our ability to collect valuable signals, especially in setups like SSAI where transparency is opaque. It then becomes a bit of a challenge for offering effective protection of the inventory.
Q: Now that White Ops has exposed this, what can the industry be doing both as publishers that have their own proprietary SSAI as well as publishers in general? What do we do from here and how do we prevent this from happening in the future?
A: It's all about securing the ecosystem. Connected TVs are an extremely attractive type of inventory for marketers. High CPMs, very engaged users, but the level of transparency and the level of security of this inventory is not adequate for its value. That's where we need to work a little bit more as an industry. We’ve made a lot of progress over the past few years but there are more things that we could do, and the first thing, especially for SSAI setups that we would like to see across the industry, is a way to verify that there is a real device sitting behind that server. We've been working with SSAI vendors and manufacturers pushing for the adoption of such standards. You may be familiar with ads.cert, which is a way of cryptographically signing requests coming out of ad servers. A similar type of standard could be implemented both on the SSAI level but also on the device level. That would give us a much higher confidence that we are talking to a real device.
Bad guys don't even need to have a real device behind the scenes because they can spoof everything. That's where we should start as an industry. We talked about the collective protection of the supply chain by having direct publisher relationships. That's why the buy side was less impacted from this operation.
We cannot stress enough the importance of adopting app-ads.txt and sellers.json. The vast majority of the inventory that we saw didn't have their suppliers declared in the respective sellers.json files.
We encourage anyone in the community to send tips they may have, including those that are not currently partnered with us. We prefer Keybase for its end-to-end encryption as well as several public key proofs that you're talking to the White Ops security research team. You can submit a type a few different ways:
1. DM us on Twitter
2. Email us at email@example.com
3. Send us a note on Keybase
For any additional questions or concerns about the ICEBUCKET investigation, please email firstname.lastname@example.org.
Watch the ICEBUCKET AMA webinar from which most of these answers came: