Tag: Asked

In the summer of 2021, President Joe Biden summoned the CEOs of the nation’s biggest tech companies to the White House.

A series of cyberattacks linked to Russia, China and Iran had left the government reeling, and the administration had asked the heads of Microsoft, Amazon, Apple, Google and others to offer concrete commitments to help the U.S. bolster its defenses.

“You have the power, the capacity and the responsibility, I believe, to raise the bar on cybersecurity,” Biden told the executives gathered in the East Room.

Microsoft had more to prove than most. Its own security lapses had contributed to some of the incursions that had prompted the summit in the first place, such as the so-called SolarWinds attack, in which Russian state-sponsored hackers stole sensitive data from federal agencies, including the National Nuclear Security Administration. Following the discovery of that breach, some members of Congress said the company should provide better cybersecurity for its customers. Others went further. Sen. Ron Wyden, a Democrat who chairs the Senate’s finance committee, called on the government to “reevaluate its dependence on Microsoft” before awarding it any more contracts.

In response to the president’s call for help, Microsoft CEO Satya Nadella pledged to give the government $150 million in technical services to help upgrade its digital security.

On the surface, it seemed a political win for the Biden administration and an instance of routine damage control from the world’s largest software company.

But Microsoft’s seemingly straightforward commitment belied a more complex, profit-driven agenda, a ProPublica investigation has found. The proposal was, in fact, a calculated business maneuver designed to bring in billions of dollars in new revenue, box competitors out of lucrative government contracts and tighten the company’s grip on federal business.

The White House Offer, as it was known inside Microsoft, would dispatch Microsoft consultants across the federal government to install the company’s cybersecurity products — which, as a part of the offer, were provided free of charge for a limited time.

But once the consultants installed the upgrades, federal customers would be effectively locked in, because shifting to a competitor after the free trial would be cumbersome and costly, according to former Microsoft employees involved in the effort, most of whom spoke on the condition of anonymity because they feared professional repercussions. At that point, the customer would have little choice but to pay for the higher subscription fees.

Two former sales leaders involved in the effort likened it to a drug dealer hooking a user with free samples. “If we give you the crack, and you take the crack, you’ll enjoy the crack,” one said. “And then when it comes time for us to take the crack away, your end users will say, ‘Don’t take it away from me.’ And you’ll be forced to pay me.”

If we give you the crack, and you take the crack, you’ll enjoy the crack. And then when it comes time for us to take the crack away, your end users will say, ‘Don’t take it away from me.’

The company, however, wanted more than those subscription fees, former salespeople said. The White House Offer would lead customers to buy other Microsoft products that ran on Azure, the company’s cloud platform, which carried additional charges based on how much storage space and computing power the customer used. The expectation was that the upgrades would ultimately “spin the meter” for Azure, helping Microsoft take market share from its main cloud rival, Amazon Web Services, the salespeople said.

In the years after Nadella made his commitment to Biden, Microsoft’s goals became reality. The Department of Defense, which had resisted the upgrades for years due to the steep cost, began paying for them once the free trial ended, laying the groundwork for future Azure consumption. So did many civilian agencies. The White House Offer got the government “hooked on Azure,” said Karan Sondhi, a former Microsoft salesperson with knowledge of the deals. “And it was successful beyond what any of us could have imagined.”

But while Microsoft’s gambit paid off handsomely for the company, legal experts told ProPublica the White House Offer deals never should have come to pass, as they sidestep or even possibly violate federal laws that regulate government procurement. Such laws generally bar gifts from contractors and require open competition for federal business.

Accepting free product upgrades and consulting services collectively worth hundreds of millions of dollars is “not like a free sample at Costco, where I can take a sample, say, ‘Thanks for the snack,’ and go on my merry way,” said Eve Lyon, an attorney who worked for four decades as a procurement specialist in the federal government. “Here, you have changed the IT culture, and it would cost a lot of money to go to another system.”

Microsoft defended its conduct. The company’s “sole goal during this period was to support an urgent request by the Administration to enhance the security posture of federal agencies who were continuously being targeted by sophisticated nation-state threat actors,” Steve Faehl, the security leader for Microsoft’s federal business, said in a statement. “There was no guarantee that agencies would purchase these licenses,” and they “were free to engage with other vendors to support their security needs,” Faehl said.

Pricing for Microsoft’s security suite was transparent, he said, and the company worked “closely with the Administration to ensure any service and support agreements were pursued ethically and in full compliance with federal laws and regulations.” Faehl said in the statement that Microsoft asked the White House to “review the deal for antitrust concerns and ensure everything was proper and they did so.”

The White House disputed that characterization, as did Tim Wu, a former presidential adviser who told ProPublica he discussed the offer with the company in a short, informal chat prior to the summit but provided no signoff. “If that’s what they’re saying, they’re misrepresenting what happened on that phone call,” he said.

A current White House official, in a statement to ProPublica, sought to distance the administration from Microsoft’s offer, which it had previously heralded as an “ambitious” cybersecurity initiative.

“This was a voluntary commitment made by Microsoft … and Microsoft alone was responsible for it,” the White House official said in the statement. Furthermore, they said the decisions to accept it were “handled solely by the respective agencies.”

“The White House is not involved in Agency decisions regarding cybersecurity and procurement,” the official said.

The official declined to comment on the legal and contracting concerns raised by experts but noted in the statement that the White House “is broadly concerned” about the risks of relying too much on any single technology vendor and “has been exploring potential policy steps to encourage Departments and Agencies to diversify where there is concentration.” Cybersecurity experts say that such concentration can leave users vulnerable to attack, outages or other disruption.

Yet the White House summit ushered in that very type of concentrated reliance, as well as the kind of anticompetitive behavior that the Biden administration has pledged to stamp out. Former Microsoft salespeople told ProPublica that during their White House Offer push, they advised federal departments to save money by dropping cybersecurity products they had purchased from competitors. Those products, they told them, were now “redundant.” Salespeople also fended off new competitors by explaining to federal customers that most of the cybersecurity tools they needed were included in the upgraded bundle.

Today, as a result of the deals, vast swaths of the federal government, including all of the military services in the Defense Department, are more reliant than ever on a single company to meet their IT needs. ProPublica’s investigation, supported by interviews with eight former Microsoft employees who were involved in the White House Offer, reveals for the first time how this sweeping transformation came to be — a change that critics say leaves Washington vulnerable, the very opposite of what Biden had set out to achieve with his summit.

“How did Microsoft become so pervasive of a player in the government?” said a former company sales leader. “Well, the government let themselves get coerced into Microsoft when Microsoft rolled the stuff out for free.”

“Everything That We Do Is Designed to Generate a Return”

The federal government is one of Microsoft’s largest customers and “the one that we’re most devoted to,” the company’s president, Brad Smith, has said. Each day, millions of federal employees use the Windows operating system and products like Word, Outlook, Excel and others to write reports, send emails, analyze data and log on to their devices. But in the months before Biden’s summit, the SolarWinds hack put that relationship to the test.

Discovered in late 2020, SolarWinds was one of the most damaging breaches in U.S. history and underscored the federal government’s vulnerability to a state-sponsored cyberattack.

Authorities established that Russian hackers exploited a flaw in a Microsoft product to steal sensitive government documents from the National Nuclear Security Administration and the National Institutes of Health, among other agencies. What they didn’t know, as ProPublica reported in June, was that one of the company’s own engineers had warned about the weakness for years, only to be dismissed by product leaders who were fearful that acknowledging it would undermine the company’s chances of winning a massive federal cloud computing contract.

But Microsoft’s known involvement was enough for Congress to summon Smith to testify in February 2021. Lawmakers focused on how Microsoft packaged its products into tiers of service — with advanced security tools attached to only the most expensive license, known to government customers as the G5.

At the time, many federal employees used a less expensive license known as the G3. As a result, they didn’t have access to the security features that might have alerted them to an intrusion and aided subsequent investigations.

Some lawmakers, like then-Rep. Jim Langevin of Rhode Island, accused the company of unfairly up-charging customers for what they considered to be basic security. “Is this a profit center for Microsoft?” he asked Smith during the hearing.

Smith replied: “We are a for-profit company. Everything that we do is designed to generate a return, other than our philanthropic work.”

Amid the criticism, Microsoft soon announced that it would provide federal customers with a “one-year free trial of Advanced Audit,” a tool that could help the government detect and investigate future attacks. Over the months that followed, Microsoft was “surprised there was not as aggressive of an uptake of Advanced Audit” as the company had wanted, Faehl, Microsoft’s federal security leader, told ProPublica. It would be a “lesson learned” going forward, he said.

That May, Biden signed an executive order requiring federal agencies to bolster their cyber defenses, declaring that “protecting our Nation from malicious cyber actors requires the Federal Government to partner with the private sector.” He added, “In the end, the trust we place in our digital infrastructure should be proportional to how trustworthy and transparent that infrastructure is, and to the consequences we will incur if that trust is misplaced.”

“Parting of the Red Sea”

Around that time, Anne Neuberger, a White House deputy national security adviser, called Smith and requested that Microsoft develop an initiative to announce at Biden’s White House cybersecurity summit that August. Like Langevin, the administration believed that the company’s advanced suite of cybersecurity tools, including ones intended to counter threats on user devices, should be included in the government’s existing licenses and that products should be delivered to customers with the most secure settings enabled by default. (Neither Neuberger nor Smith granted interview requests.)

Giving away a bundle of advanced security features permanently was a nonstarter inside Microsoft, an executive told ProPublica. But Smith spearheaded a team to develop an offer that appeared to be a compromise.

Federal customers could have free, limited-time access to the upgraded G5 security capabilities and to consultants who would install them. “It was at the behest of the Administration that Microsoft provided enhanced security tools, at no cost, to agencies as soon as possible to level up their security baseline,” Faehl told ProPublica.

While the deal achieved the administration’s goal of better protection for the federal government, it also served Microsoft’s interests. Microsoft salespeople had been trying, unsuccessfully, for years to convince federal customers to upgrade to the G5. Department and agency officials balked at the higher price tag when they already had other vendors providing some of the same security capabilities. The G5’s retail price is nearly 60% more than the G3’s.

“We knew that this was a golden window that nobody could have foreseen opening up because we had been pushing” for the G5 upgrade “for years, and things were going very slow,” said a former Microsoft sales leader involved in the strategy. With the White House Offer, it was “like Moses leading us through the parting of the Red Sea, and we just rushed through it.”

We knew that this was a golden window that nobody could have foreseen opening up.

Faehl told ProPublica that sales of the G5 had been slow prior to SolarWinds because federal customers wrongly believed “that they had sufficient security capabilities already in place.” He said the attack was “a wakeup call showing the status quo perspective to be insufficient.”

Microsoft was well aware of the possible legal implications of its offer. More than two decades ago, the U.S. Department of Justice sued the company in a landmark antitrust case that nearly resulted in its breakup. Federal prosecutors alleged that Microsoft maintained an illegal monopoly in the operating system market through anticompetitive behaviors that prevented rivals from getting a foothold. Ultimately, the Justice Department settled with Microsoft, and a federal judge approved a consent decree that imposed restrictions on how the company could develop and license software. Although the decree had long since expired, it nonetheless continued to loom large in the corporate culture.

When it came to the White House Offer, company insiders were “mindful of the concerns about Microsoft making products free that smaller companies sell,” an executive told ProPublica. A spokesperson explained, “That was the impetus for asking the administration to review it.”

The “review” consisted of a phone call between Microsoft’s Smith and Wu, who was Biden’s special assistant for technology and competition policy.

“Brad was like, ‘We think security is important, and we want to give the federal government better security,’” Wu recalled.

But, according to Wu, Smith said Microsoft’s lawyers were “overly paranoid” about antitrust concerns, and he was looking to “calm his own lawyers down.”

“I made it clear there was no ability in the White House to sign off on antitrust,” which is in the purview of the Justice Department or the Federal Trade Commission, Wu said. “I’m smart enough not to say, ‘Oh yeah, that’s fine with me.’ I’m not crazy.”

I made it clear there was no ability in the White House to sign off on antitrust. I’m smart enough not to say, ‘Oh yeah, that’s fine with me.’ I’m not crazy.

After the news organization asked Microsoft about Wu’s account, a spokesperson walked back the company’s original written statement, saying that Faehl was misinformed. “The White House arranged a call and we described details of our security offer and how it was structured to avoid antitrust concerns,” the spokesperson said. “It was an informal conversation and at no time did we ask for formal antitrust approval.”

Wu also told ProPublica that he felt pressure from the National Security Council’s Neuberger, who “wanted to get this deal done” in the wake of SolarWinds and other cyberattacks. “She pushed me to get on the phone with Brad,” he said. “I feel in some ways in retrospect I should not have even spoken with him. But I felt that I should help the NSC for what they presented as a formalistic exercise to help the national security.”

“The End Game”

After the White House summit, Microsoft’s sales teams quickly mobilized to sell the “WHO,” as it became known to insiders. The free consulting services were a crucial part of the strategy, former salespeople said. As Sondhi put it, “Just because you give the product away for free doesn’t mean they’re going to use it because it’s a pain in the ass to install new software and retrain staff.” The company wanted to avoid a repeat of the disappointing participation in the earlier Advanced Audit offer.

The consultants would work inside the agencies, where they would have government-provided desks, phones and internet, as well as access to federal computer networks, according to one proposal reviewed by ProPublica. From their perches in the bureaucracy, they would get the products up and running and train federal employees on how to use them. This would make the upgrades “sticky,” as they became ingrained in employees’ daily lives, former salespeople said.

Microsoft covered the free product upgrades for up to a year, the company told ProPublica. Faehl called the free upgrades “a short term option for protection while agencies put long term procurement plans in motion.” Or, as sales teams told customers, they “should not have to wait to be secure until they can procure.” The company also noted the offer came at a significant cost to Microsoft, “with no guarantee of renewal once the deal expired.”

But sales teams said they knew customers who accepted the White House Offer were unlikely to undo the intensive work of installing the upgrades when renewal time rolled around, locking them into the G5 for the long haul. Wes Anderson, a Microsoft vice president who oversaw teams working with the Defense Department, asked his staff to prepare forecasts showing which customers were expected to become paying G5 users at the end of the White House Offer, three people who worked in sales told ProPublica.

“It was explicit that this was the end game,” one former Microsoft sales leader who worked inside the Defense Department told ProPublica. “You will do whatever you need to do to get that software installed, operational and connected so the customer has their runway to renew.”

It was explicit that this was the end game. You will do whatever you need to do to get that software installed, operational and connected.

(On Oct. 30, two weeks after the news organization sent Microsoft questions for this story, the company announced in an email to employees that Anderson would be leaving Microsoft. Neither Anderson nor Microsoft commented on the departure. On the topic of Anderson’s request of his staff, the company said, “Forecasting is part of the rhythm of business for organizations in nearly every industry.”)

Salespeople pitched the White House Offer as “the easy button,” people familiar with the strategy told ProPublica. “Our argument was, ‘We have this whole suite of goodness,’” said a former Microsoft employee who worked with the Department of Defense. “‘You should upgrade because it will take care of everything rather than having a bunch of vendors that each do one of the 20 things that the G5 can do.’” Faehl told ProPublica the license bundles help federal customers “avoid the hassles of managing multiple contracts and licenses” and close security gaps by replacing a “patchwork of solutions” with “simplified, comprehensive protection.”

For the most part, as they predicted, the Microsoft sales teams found receptive audiences across the government. To help ingratiate themselves, they invoked their association with the White House in their pitches. In one example, from June 2022, a Microsoft representative wrote to Veterans Affairs officials to explain that, “working in conjunction with the White House,” it would provide “a no cost offer of professional services to provide hands-on assistance” to deploy the upgrades.

Money for Nothing?

As consultants fanned out across the federal government to turn on the new features, there was a sense of unease among some employees about the nature of the deals. Typically, the government obtains products and services through a competitive bidding process, selecting from a variety of proposals from different vendors. The White House Offer was different.

“No matter how you wanted to polish the turd, there was the appearance of no-bid contracts,” said a former Microsoft consultant involved in the WHO.

The federal government may receive so-called gratuitous — or free — services from donors as long as both parties have a written agreement stating that the donor will not be paid for the goods or services provided. Such agreements were in place for the consulting services in the White House Offer, the company said.

No matter how you wanted to polish the turd, there was the appearance of no-bid contracts.

Those agreements may have helped Microsoft pass the “laugh test,” said Lyon, the former federal procurement attorney. “But just because something is technically legal does not make it right,” she said.

Other contracting experts said federal departments and agencies should have been more skeptical about accepting free products and consulting services from Microsoft, given the implications for competition and national security.

The cost and difficulty of switching from the Microsoft products presents a classic example of “vendor lock-in,” said Jessica Tillipman, associate dean for government procurement law studies at George Washington University Law School. “The free services are allowing the government to bypass a competitive procurement process and locking them in for future procurements,” she said.

Tillipman said that, in the future, the government should consider restrictions on gratuitous services in IT in order “to ensure you’re not locked in with a vendor who gets their foot in the door with a frighteningly expensive” giveaway.

“This is all designed to undermine future competitions,” she said.

This is all designed to undermine future competitions.

James Nagle, a former Army contracting official and practicing attorney who specializes in the federal contracting process, went even further, saying that the White House Offer potentially violated existing law.

The Federal Acquisition Regulation, which governs government procurement, says that employees may not accept “gratuities,” or anything of value “from anyone who has or is seeking to obtain Government business.” And, as employees involved with the White House Offer told ProPublica, Microsoft was seeking future contract upgrades and new Azure revenue.

While “gratuities” are typically considered to be perks such as free meals, sports tickets or other gifts for personal use, Nagle argued that the rule could apply to the White House Offer, though he said he was not aware of any prior case using his interpretation. He compared it to a car manufacturer providing a government agency with a fleet of cars for a year for free because it wants the agency to procure that fleet for its staff. “Any contracting officer would say, ‘No, you can’t do that,’” Nagle said. Once employees get used to the cars, they’re reluctant to switch, he said, and the “impermissible gift” would create a built-in bias toward that manufacturer.

“That’s the problem here,” Nagle said. “This is not truly gratuitous. There’s another agenda in the works.”

Microsoft did not use the so-called gratuitous services agreements to give away the G5 upgrades, as it did for the consulting services. Instead, Faehl told ProPublica, the company considered them “a 100% discount” added to existing customer contracts. He said making this type of “strategic investment is … common practice among companies” and that contract teams on both sides reviewed the deals. Nagle viewed it differently, characterizing the free products as a “loss leader designed to lead to future sweetheart deals.”

Federal vendors may be banned from government contracting for violating the Federal Acquisition Regulation, though such an outcome would be highly unlikely for a vendor as large as Microsoft, Nagle said. Nonetheless, individual employees on both sides of improper deals in the past have been held accountable, he said.

Skirting fiscal law, however, may have set the stage for an even more serious legal concern, said Christopher Sagers, a professor of antitrust law at Cleveland State University in Ohio. Microsoft’s actions, Sagers said, might constitute what is known in antitrust law as “exclusionary conduct,” opening the door for illegal monopoly. “Microsoft, rather than competing on the merits, took steps to exclude competitors by making its product sticky in advance of opportunities for competition,” he said. The company used “an already dominant position to further cement their position.”

Microsoft disputed that point.

“We don’t believe our offer raised antitrust concerns, and we constructed it specifically to avoid any such issues,” a company spokesperson said. “We talked informally with a White House staffer about this.”

Wu, however, said the company did not make clear to him the financial and competitive implications of the offer.

“There is no way that was discussed,” Wu told ProPublica. “The only thing that Brad mentioned was upgrading federal agencies, offering them better stuff.” Upon hearing the news organization’s findings, he said: “That is a lot darker than it sounded. Once you’re in somewhere, it’s very hard to leave.

“Now I’m starting to feel guilty in some weird way about playing a role in a big deal that cost taxpayers money,” Wu said.

Taking Out the Competition

Former Microsoft salespeople said that all of the customers within the Defense Department who signed on to the White House Offer — including all the military branches — ultimately upgraded to the G5 and began paying for it when the time came to renew their agreements in 2022 and 2023.

A Defense Department spokesperson said in a written statement that the department followed federal acquisition law and “conducted internal tests and evaluations of multiple vendor capabilities.” The upgrade, the spokesperson said, was “crucial” to meeting the department’s cybersecurity objectives. The department declined to answer follow-up questions, including to specify which vendors it evaluated before deciding on the G5.

John Sherman, the department’s chief information officer at the time of the White House Offer dealmaking, defended both the government’s decision and Microsoft’s strategy. “I am sure Microsoft, like any company, would be trying to increase their business with any customer,” he told ProPublica.

He added, “We didn’t have any particular preference for Microsoft in terms of favoritism or anything like that, but we knew it worked, which is why we wanted to proceed with that.”

Many civilian agencies also upgraded to the G5 during this timeframe, said Sondhi, who now works at Microsoft competitor Trellix as chief technology officer for the company’s public-sector business.

For Microsoft, winning more government business was only half the picture. It also saw the White House Offer as an opportunity to knock out its competitors.

During and after their sales push, Microsoft salespeople advised government departments and agencies to remove competing products from their IT lineups to cut costs, saying the Microsoft bundle would render those other products redundant. Internally, employees called it the “take-out” strategy. “The play is: ‘You’re paying for it in the G5. It’s a waste of government money to have both,’” a former sales leader who worked with the Defense Department told ProPublica.

Sondhi said that in a typical scenario, an upgrade to the 5-level can displace the existing work of a half dozen vendors or more. Executives from cybersecurity companies Trellix and Proofpoint, for example, told ProPublica they lost federal business in the wake of the White House Offer deals.

The White House Offer also enhanced Microsoft’s competitive position by reducing the likelihood that the government would open bidding for cybersecurity products in the future, given the cornucopia of offerings in the G5. Within the company, this was known as “taking opportunities off the street,” former sales leaders said.

The fallout impacted companies that were in the midst of completing the authorization process the government requires of vendors providing cloud-based services. Several told ProPublica that cybersecurity contract opportunities are now scarce.

“We are chipping away, but it’s largely, by far, a Microsoft-owned landscape,” an executive at one competing vendor told ProPublica.

Faehl dismissed those complaints, saying that customers kept the upgrades because they performed well and that competitors “should look inward to see why their products do not meet or exceed Microsoft results.”

Reckoning With the “Monoculture”

Microsoft has something few other companies possess: a panoply of products that span the IT ecosystem. Rivals say the company leveraged its existing dominance in certain products — like the Windows operating system and classic workplace applications — to gain dominance in others, namely cybersecurity and cloud computing.

“No one has the kind of capital that Microsoft does,” Sondhi said. “They can just absorb the cost of the giveaway until the customer’s first bill.”

A coalition backed by some of Microsoft’s major competitors, including Google and Amazon, has raised similar issues with the Federal Trade Commission, which in 2023 gathered public comments on the business practices of cloud computing providers. Among the FTC’s areas of ongoing interest: “Are there signs that cloud markets are functioning less than fully competitively, and that certain business practices are inhibiting competition?”

Competition is not the only issue at stake. As Washington has deepened its relationship with Microsoft, congressional leaders have raised concerns about what they call a cybersecurity “monoculture” in the federal government. Some, like Wyden and Sen. Eric Schmitt, a Republican from Missouri, have blasted the Defense Department in particular for “doubling down on a failed strategy of increasing its dependence on Microsoft.”

“Although we welcome the Department’s decision to invest in greater cybersecurity, we are deeply concerned that DoD is choosing not to pursue a multi-vendor approach that would result in greater competition, lower long-term costs, and better outcomes related to cybersecurity,” the two lawmakers wrote in a letter to Sherman, then the department’s chief information officer, in May.

Microsoft’s Faehl pushed back. “The suggestion that our customers are any more at risk because they use Windows, or Azure, or Office is wrong,” he said. “We partner closely with our security competitors because we see them as partners against threat actors we face in common.”

Still, just last year, Chinese hackers exploited Microsoft security lapses to breach the email accounts of senior U.S. officials. Investigating the attack, the federal Cyber Safety Review Board faulted the company for a “cascade of … avoidable errors” and pressed it to overhaul its security culture. Microsoft has since pledged to place security “above all else.” In June, Smith told Congress that Microsoft would strive to establish a “culture that encourages every employee to look for problems, find problems, report problems, help fix problems and then learn from the problems.”

It’s learning from the successes, too. The same week that Smith testified before Congress, and nearly three years after Nadella made his commitment at Biden’s summit, Microsoft made a new offer, this time to “support hospitals serving more than 60 million people living in rural America.”

The playbook was familiar. In its announcement, the company said that eligible hospitals could have the private-sector equivalent of the G5 “at no cost for one year.” As before, Faehl said Microsoft made the commitment “at the behest of the White House.”