A Failure of Power - What We Can Learn From Texas
The Sound of Silence
Monday morning I woke up at 2AM to the eerie sound of snow blowing off the buildings on my street. Normally, E 6th St glows with streetlamps, headlights, and window signs. The train honks its horn at all hours. Cars bump down the road playing loud reggaetón.
Instead of the live music capitol of the world, it was the utter silence of a snowy Christmas card scene. The buildings creaked with the weight of the wind and the snow.
I'd been warned the week before by my work's emergency alert system to expect 1-3" of snow through noon on Monday. When I told people at the dog park we were supposedly expecting snow they were rightly confused. I picked up a couple of extra soup cans at HEB just in case and thought, a year ago you couldn't find a single can in sight. The COVID-19 pandemic was coming around the corner and now a year later…still in a state of emergency. But we have toilet paper this time.
With the shelves full and people going about their masked, socially distanced lives it didn’t feel like we’d be waking up to frigid houses, cold toes and faces. Or that it would continue with icicle encrusted ceiling fans. As the freeze was seeping in through the window I picked up my phone. It was calm but ghostly. I had another alert from work saying to expect rolling blackouts lasting 40 minutes or less. Two hours later I was still awake, waiting for the blackout to end.
This was just the start.
The Outage
Between then and Wednesday late afternoon, we had power on twice a day anywhere from 5 min to 2 hours. When it first came back, all my devices powered on and then in 5 min it was out again. I thought about inrush current and everyone’s devices doing the same thing. I put all my sweaters on the window seals, closed the blinds to retain heat, and set up camp on my couch… which my pizza thermometer said was 5-10 degrees warmer than my bed. A friend texted me that they could see their breath inside their apartment.
As a person without power, and a quickly dwindling cell phone charge, I craved answers. By 9AM Monday I was spinning in circles in my condo trying to decide what to pack, if I need to vacate my pets, is a main breaker out and how long would that take crews to fix?
A very confusing outage map.
There are three grids in the US: the eastern interconnection, the western interconnection, and the Electrical Reliability Council of Texas (ERCOT) which was established for the main reason of evading federal regulations, in particular the Federal Power Act in 1935. The ERCOT grid is still not subject to the Federal Power Commission which is now called the Federal Energy Regulatory Commission (FERC). ERCOT, a non-profit, is responsible for 90% of the state’s power to 26 million citizens and subject to Texas regulators known as the Public Utility Commission (PUC).
In 2011 (we’ll get to this later) Texas was able to pull in power supply from Mexico. The same Mexico we’ve just politically, literally, and figuratively build a wall between them and Texas. Today ERCOT has three ties to Mexico and two ties to the eastern grid. That gets them out of regulations from feds. My coworker reminds me that Texas is the only state that’s a state by Constitution, only state allowed to fly at flag at the same height as the US flag, and retains the ability to secede. In short, Texas is a separatist and it’s definitely about political power.
“In the name of deregulation and free markets, critics say, Texas has created an electric grid that puts an emphasis on cheap prices over reliable service.” Washington Post
The biggest concern is if there is too much power draw it can overload the system and pull the whole grid down. Texas was apparently “seconds or minutes” from this catastrophe happening. The issue then is that they would only be able to power up the black start equipment and it would have been days of no power across Texas. Even after the grid was restored, there were still pockets without power due to broken equipment. It’s not hard to imagine that the older equipment was most impacted, which is potentially not equitably resourced.
Polar Vortex
It was colder in Texas than it was in Alaska this week. Dallas had the coldest temperature on Tuesday since 1903 with a low of -2F. Austin and San Antonio also beat out old records from 1900-1910.
I cooked breakfast by headlamp (sort of surprised I had batteries) and the gas range (same to the lighter). My neighbors a building over offered some pre-ground coffee beans. I poured hot water over the grounds, tea-style. I put on a few pairs of socks and rainboots. Remember this is 100-days-over-100F-degrees country. Buying winter boots was never on my to-do list.
From WP: The barrage of winter storms and onslaught of Arctic cold are the product of particularly placed weather systems, a disrupted polar vortex and a pattern favoring cold air arriving from the high latitudes. https://www.washingtonpost.com/weather/2021/02/16/winter-storm-live-updates/
My neighbors, the ones that hadn’t left for friends’ houses (amid a global pandemic that took second priority to many of us), were bunkered in. They used their camping & skiing gear. They charged their phones in their cars. I do not have a car so Harley and I started to trek to my brother’s house, her jumping out of the 6” snow. Me struggling to keep hold of her leash. Everyone I passed I said, “You got power?” Nothing. From anyone. Some student-aged kids told me they didn’t have any way to make food. A guy in a winter jacket with a lift ticket told me he was just going for a walk. This part of town is gentrifying fast (and that’s a whole other equity discussion).
We trudged through the fresh snow, it seeping into my rainboot tops. My phone was almost dead so I regrettably don’t have pictures of people skiing down 12th St or the fresh powder we trudged through on Pleasant Valley. We walked in the middle of the quiet streets where a few tracks had been made by 4WD Texan trucks. It was beautiful in a serene, otherworldly way and I wish I had been in the state of mind to capture it. The puppy loved it until we walked over an ice&snow covered bridge with wind billowing at us. 10F and it felt like we were in the arctic.
We were calling this palm the “horton hears a who death tree” as every time the wind blew, shards of ice peppered the ground.
Around 430PM Harley and I hiked back. My neighbors texted that we got power! Before I made it home, power was off again and my right foot was bruised from the boots I’d chosen. We hung out in the dog park a while: nobody had anything else to do. And then I read by electric candlelight on our makeshift couch campsite. Tuesday morning, still no power, I found some different steel-toed work boots to make the trek again to charge my phone. I did some work and checked in with my people.
Deja Vu
In 2011, over 300,000 Texans woke up without power. It knocked hospitals offline but kept the Cowboys stadium lit up. The most vulnerable suffer while those with money are unaffected (and sometimes oblivious to what it means to be a tale of two cities). This is systematic racism and once we identify it we should put in proper processes to make sure our designs & our emergency plans are equitable. The root cause analysis from 2011 is available here. The recommendations include winterization of equipment and better communications.
In 2021, a total of 4.5 million people were without electricity on Tuesday alone. Downtown Austin, including the stadium, vacant government buildings due to the pandemic, and high-rise under-construction skyscrapers were lit up. Next door in the historically underprivileged, black and brown part of town known as East Austin was pitch black for multiple days in subfreezing weather. The blackouts were not rolling.
Dishearteningly, we obviously did not prioritize the inequities made obvious in 2011.
East Austin on the left, downtown on the right, separated by I-35
Even with a major telephone carrier down we all saw this photo. Austin Energy released a statement after this photo circulated. They said that Austin downtown was excluded from the ERCOT mandated load shedding due to “a part of a critical network of infrastructure which includes buildings such as the Dell Seton Medical Center, warming centers, the COVID-19 Alternate Care Site, the Capitol Complex and Austin City Hall and other government buildings”. The part of town that was pitch black and frozen was not the part of town that would have financial resources for winter gear and supplies.
The plan was to do 40 min rolling blackouts. Just like my work text said. In actuality that’s not what happened. Which means that this was planned out for a different scenario than what actually happened. Decisions were made quickly but that still hurt BIPOC families. Some complained that the power decisions being made were to save ERCOT money and there was lots of finger pointing even from officials.
Apparent negotiations started between the big power users downtown, the big industrial users up north and east and Austin Energy who reached out to 800 properties on Tuesday. While this may not have been the intention of the emergency plan, the actuality was a giant manufacturer had power the two days that the historically underprivileged east side had nothing. Even with changes to the grid on Wednesday, many on the east side were still in the dark.
In the dark without power OR communications.
Cities need to revisit all the emergency management plans and do simulations of power and water loss to identify risks to the system and social biases. And furthermore, come up with a robust and equitable load shedding plan that’s ready for the most dire situations.
State of Power
Political Power
State and federal declarations of emergency were issued for all 254 Texas counties and FEMA mobilized. Tuesday night, my power came back on. Since it’d come on for a couple of times for not very long, I was not confident it would stay. The next day, the governor signed an executive order directing natural gas providers to redirect all out-of-state gas shipment to stay in Texas. “Every source of power Texas has has been compromised,” Mr. Abbott said, from coal and renewable energy to nuclear power.
31 people died from the cold, from trying to stay warm and driving on unprepped roads in Texas. This was a week after the deadly 133 car pile-up in Ft Worth. A 70-yr old man died in his bed from the cold. As I write this, people still do not have power in some places (Saturday update: power is back!). Vaccine distribution lines were disrupted. St. David’s hospital relocated 300 patients when they didn’t have water pressure to run the boilers to power the building. The Mayor of Austin in a chilling briefing: “If you have power, please try to live almost like you don’t. If you have heat, run it low. Run it lower.” Individual people were being asked to take on the burden of conservation while large power users continued operations for two days.
Senator Beto set up a call center and made 151,000 calls to senior citizens to deliver resources to the most vulnerable.
I have to believe we have the innovative engineering tools to set up something like this ahead of time. It needs funding and planning.
The governor called for the resignation of ERCOT’s leaders after lack of communications and reliability. Last week he had been assured that they were prepared. He also spread some misinformation on the state of renewables during that interview. Many other leaders jumped in to push their agendas against renewables leading to lots of false information.
News briefing via zoom
Energy Power
In 2011, 8,000MW went down, about 12% of the state’s power supply. In this week’s outage, almost six times that amount (46,000MW) went down from 185 generating plants. Generators which tie into ERCOT come from many different generation types. We even have two nuclear reactors which are owned by Austin Energy, CPS, and NRG.
Normal ERCOT power sources
By Tuesday, natural gas was 40% of our generation and the second largest generator was wind at 23%. As of Wednesday, we were getting about 40% from solar&wind and 60% from coal&gas (18,000MW and 28,000MW respectively). All these numbers to say that we primarily lost generation from the non-renewable sources. Less than 13% of the outages were from wind, even though Fox News ran a narrative that renewables were to blame.
So let’s look at the power sources at play here.
Wind - Some blades did freeze up and those were splashed on headlines; this renewable is generating power when the wind blows so even though it generally operates at 25% of the mix there would never be a scenario where we would rely only on this source
Solar - Similarly, this renewable generates only during the daytime but is less efficient on cloudy, arctic days so it too has to be part of the mix but not the full solution
Coal/gas - These undergo routine maintenance in the winter and are sometimes offline; it also costs money to winterize them so that is a cost sometimes skipped by private generation companies in Texas
Nuclear - Clean energy but doesn’t always have public appeal
Biomass - Looks like it was mothballed for the winter months and is more expensive to operate than other sources
Pumped hydro - This is when you pump water uphill at night when energy is cheaper (supply and demand laws) and let it flow downhill through the turbines during the day when energy is worth more; it’s not creating energy but is optimizing it
Hydroelectric - Releasing water through dams to run turbines; these need elevation differences which we have in TX but dams in waterways are an environmental concern
Hydrogen - Assumed not in our mix but this is a power opportunity getting a lot of buzz right now in the industry
Resilient and sustainable
We saw this happen before so this predictable event should have been in our scenario planning.
If we’re going to be resilient, we need a governance to generators in the state of Texas for winterizing and coordinating better. And we need to look into what is causing the polar vortex in the first place - these same fossil fuel driven energy systems.
“Our planning is based on outdated weather patterns, and if you use outdated weather, you never expect to freeze,” said Michael Webber, an energy resources professor at the University of Texas at Austin.
By the way, his book Power Trip is one of my favorites on energy and will explain this all much better that I could.
Water Woes
Each night we had a hard freeze. During the day the roads were passible but at night it became dangerous again. Five frozen nights later, the ice is finally melting from its thick icicles coating the surfaces. But with rising temperatures means that the frozen pipes had expanded and busted. The older houses without renovations (think marginalized communities) have copper pipes which are more susceptible to cracking than PEX pipes. We even have cast iron pipes from the 1930s. By Thursday, 13 million Texas had impacted water quality. I walked outside to water coming out of the seep holes of the building. Water was squirting out of the encasement around a building drainage pipe. It was coming out from under the door to the sprinkler room.
I made a couple of calls and as a neighbor came over to shut off the valve (cursing that the valve was frozen open) I ran upstairs calling my neighbors and banging on doors, “Fill your tub! We’re shutting off the water!” Austin Water has posted that water use was up 250% and to conserve water. We were losing 325 million gallons of water at the peak. I had mine off for 24 hours (a fast turnaround to get a plumber out on site). Again, individuals are being asked to take on the burden of conservation even though there are large water users on the system.
Smart metering can help with this by identifying leaks much sooner than relying on someone to call 311. Smart meters should be prioritized in equitable ways.
Many are still without water on Saturday as two of our three pressures zones are down. A boil water notice was issued for low water pressures, something that occurs when storage is depleted and system pressure is less than 35 psi. This is due to the ability of groundwater to seep into less pressurized pipes known as inflow. Seven million Texans went under boil water notice on Wednesday, not counting those of us with no water.
From Austin Water operational updates site
I’ve been responding to friends about what to do in a boil water notice, how to flush a toilet with a bucket of water, when to stream or drip a faucet. Sprinkler lines were not winterized ahead of time and judging by the HEB shelves people weren’t prepared.
An awning fell off one of the buildings, loaded up with ice and our board went around in black ice conditions trying to relieve the weight from the awnings. I somehow became a point person to check the property for burst pipes every few hours. We have to record the checks because the fire suppression system froze and burst. This has been a constant thing trying to keep people and animals safe in Austin. It took a disaster before people understood how our grid works and the implications for a state-wide blackout.
We don’t have good public education around emergency failures so that people can make informed decisions to keep their families safe.
What we need to do differently
My experience wasn’t nearly the worst of it. I feel privileged to have gotten a hot shower. The storm affected everyone but the infrastructure failures disproportionately impacted different demographics. As engineers and stewards of public trust it is our responsibility to better design our systems now that we know the weak spots. I listened to a news briefing and heard officials call it a “success” that we did not lose the grid with the blackouts. While that may be true, it is tone deaf to a lot of Texans’ experiences and the 30 deaths in Texas to hypothermia, carbon monoxide poisoning, and icy road conditions.
More standardized communications network needed
Friday morning I got my first email with a link to the outage map. If companies can buy my personal data to send me junk mail surely we can set up some official communications. We were distinctly unprepared as a community and as individuals. We relied on people calling into 311 and then reporting out on social media for information. One of the major phone carriers went down and had many people without phone or internet. Let’s update those communication systems and make them redundant to issues like this. Companies need to partner together to ensure reliability for our most vulnerable.
Hours long lines at HEB in subfreezing weather. I’d walked here twice before to find it not open. By Thursday there was no fresh food and no refrigerated food. My neighbors waited in line at a different store and grabbed me some fresh food and a can of beans. They’re the best.
More funding for infrastructure needed
We knew that our infrastructure was underfunded and would affect the economy. If a utility is barely operating during daily operations, why did we expect it to be any different in an emergency?
We need to be strategic about the funding. Some components which can be automated, should be. Having the outage map was helpful in knowing which friends to call to check on but even the outage map we had was confusing. Smart meters can help to identify outages with greater accuracy and transparency, freeing up our people to act fast during an emergency with real-time data.
An instagram account created a crowd-sourced list of everything open around Austin. This was our bible to free food and resources, though wasn’t always accurate. Others also made geospatial resource maps to help. Things like this can be innovatively automated, shaped by our geodigital tools, to come online during an emergency. We need education and training on this so volunteers can respond without reinventing the same things.
A standard rule of thumb for resiliency is that for every $1 spent in disaster mitigation, saves $6 down the line. It is easier and cheaper to build equipment to sustain winter conditions than to retrofit it. We need a plan to build-back-better as we routinely decommission old equipment. I have this guide on resilient design written in plain terms.
prioritize social-based scenario planning
I’ve talked before about how design is not inherently unbiased. We have the capabilities to do social impact based analysis of design before a single dollar is spent. We also have capabilities to do power and water loss simulations with an eye to social impact. This is not something always prioritized by utilities that struggle with funding to replace aging infrastructure but 2021 has made it obvious that this is our responsibility to people on the margins. We need diverse players in this space, including charities, that can raise funds.
More connections and oversight by federal level
The Public Utilities Commission of Texas has ordered that ERCOT “update energy prices to reflect the demand”. To put this in perspective, prices soared from $22/MW to $9,000/MW while 4 million Texans were without power. The PUC is the agency that regulates ERCOT as it is not federally regulated. FERC made recommendations in 2011 but with no federal oversight were not followed.
More connections between the power grid may mean that we would have had rolling brownouts instead of sustained blackouts. Analysis is being done to look into a southwest connection, reports Scientific American. Texas may even make money from that connection which leads me to my next point.
Reparations
There will be an investigation next week into ERCOT and PUC along with talk of financial aid to private companies to winterize their equipment. There’s going to be lots of talks by economists and numbers-people. While there is money on the negotiating table, reparations should be a part of the conversation.
Reading: by electric candlelight
Working: to share what’s happening here in Texas
Listening: the birds digging through the snow on my balcony for seeds