250MW of Solar Power - My Time at Clir Renewables

David

Introduction

“I kept dreaming of a world I thought I’d never see. And then, one day…”

I cringe to look back on that moment, but receiving an offer to intern at Clir Renewables for the summer of 2021 was one of the happiest instants of my university education.

At that time, Clir was a small Vancouver-based tech startup, fresh out of a Series A round of funding, with a grand mission to provide cutting-edge data science services to every single wind, solar, and battery plant in the world. I was a miserable and overworked university student, holed up in an apartment due to pandemic restrictions, who desperately wanted IN to the renewable energy industry.

I had built up a non-trivial quantity of power systems and software experience from my work in UBC’s IEEE branch, from my contributions to the UBC Solar engineering design team, and from my past co-op experiences. I resolved to make my last co-op term count: I must get a job in renewables by May.

I contacted a former colleague who was working at Clir, and he invited me to send the company my resume. I sat the interview with several renewable energy analysts, passed it, and on a cold evening in April, I got an offer in my e-mail.

I wish that this was a story of resounding triumph, but this is more of a story of walls, fog, tunnelling, and the occasional breakthrough.

The Renewable Analytics Co-op

“…I Got In.”

I arrived at the Vancouver office(which was still closed due to pandemic restrictions), and picked up my laptop. For the first two weeks, I received the briefing on the company operations from the friendly managers of the renewable analysts’ department.

Days quickly settled into a routine. The first few hours of every week would be spent manually updating a KPI dashboard for several high-priority clients, and the first few days of every month would be spent assembling and checking over analytics reports to be delivered to clients. This part was exceptionally frustrating - there was no artistry, no elegance or efficiency, in preventing MS Excel macros and document bindings from collapsing on themselves.

“Together we’re going to change the world, man”

However, the rest of the work made up for the tedium of assembling client deliverables. As an analyst, I was also given a free reign in designing and building tools to help other analysts, now and into the future. I started from a collection of ad-hoc Python scripts used to automate the most obnoxious tasks. As the number of scripts grew, generalizations emerged across these scripts, and I consolidated them into their own modules. The end result was a kind of modular pipeline, where data is downloaded from an S3 bucket, filtered and processed in several discrete steps, outputting KPIs at the end. This buildup of technical capability culminated in a tech demo in front of the entire company, where I presented my work to wide approval.

In the meantime, the company had just closed a Series B funding round, and spirits were exceptionally high. New staff were being hired left and right. Alongside my regular duties, I sat in on meetings on the expansion of Clir’s solar product, where heavy emphasis was placed on model correctness - building on the lessons learnt from building out the wind product, we were determined to get it right the first time.

“Flynn! Am I still to create the perfect system?”

Yet, while work appeared to be going great, I began having unsettling thoughts. How impactful could my work be to the organization as a whole, when I can only access a tiny fraction of the company infrastructure and code base? How could it be, that meetings after meetings could go on without conclusion, without anyone pointing it out? Is my current pace really the fastest I could do my work(No it wasn’t - I was working fast, but my apartment internet connection at the time was horrendous and in hindsight, it was obviously the limiting factor for many data intensive tasks).

And most concerning of all, how do I know that this is the best, the most correct, the most accurate way of doing things, when my supervisors and clients were already happy with the results as they stand?

My internship came to an end, and I received an assurance of a return offer. And while I definitely learnt and contributed a lot in 4 months, in hindsight I had spent the summer in a fog of confusion. There was still much more to learn.

The Renewable Analytics Analyst

“Maximize efficiency. Rid the new system of its imperfection! My vision is clear, fellow programs. Out there is a new world! Out there is our victory!”

8 months later, with an engineering degree in hand, I returned to Clir Renewables as a full time renewable energy analyst. I picked up my laptop from the office, logged in to several welcome calls, and introduced myself to the new analysts. To pleasant surprise, the solar analytics tooling I had built was still being used actively, and had even become essential. To my dismay, there had been virtually no improvements to the tooling since I last committed to it - it was either ‘good enough’, or no one had taken the initiative to improve it.

But now, I am here and better than ever before. I yearned to take the lessons I had learnt to apply it into analytic tools development. In the words of Clu from Tron: Legacy, I resolved, in my mind, to ‘take the system to its maximum potential’.

“The game has changed, son of Flynn!”

Two days later, the layoffs started.

Teams were gutted, managers and top-level executives, removed. The colleagues who I admired and trusted as a co-op, terminated. In a company wide call and in the weeks to come, it dawned on all of us that we are now operating in a much harsher investment climate than what our systems and operations were designed for. Rising atmospheric temperatures might doom human civilization if net-negative emissions are not reached in several decades, but rising interest rates will doom Clir if net-positive cash flow is not achieved in several quarters.

“I think he’s knocking on the sky. And listening to the sound.”

I confess that I did spend a few days moping about in denial, but we all still had work to do. Again, I did my share of client deliverables, but where I really threw myself into was into improving the solar analytics tooling and novel prototyping work. New interfaces bridged similarities between solar farms, allowing for further code reuse. Obsolete code was refactored into vectorized equivalents, improving processing speeds by an order of magnitude. New anomaly detectors were prototyped, validated and submitted as feature requests. I reviewed the electrical schema and single line diagrams of every solar farm in our system, and gained insight into many different solar farm morphologies. I made several attempts at an accurate measurement of PV degradation or soiling rates on several solar farms, with mixed results - the signal-to-noise ratio simply made the problem intractable. Finally, I was able to leverage my knowledge of software development and analytics to bridge knowledge gaps between both teams.

“Tron, what have you become?”

Despite this progress, I still could not adequately address my prior concerns as an intern. I wondered if our innovative tool-building is appreciated on a larger scale. The lines of communication and collaboration within the company introduces impedance between software development and renewable energy analysts, and I was haunted by the possibility of divergence between our work and wider product and technical realities. And if our work was uncovering new realms of insights, then it frustrated me why new prototype releases were being seen as threats to reporting continuity rather than hard-fought steps towards truth.

I got what I wanted, but it wasn’t what I needed. Unlike last time, I could see the fog of confusion, and this was not where I needed to be.

Looking back, I wish that I had understood the process of burnout better. Client deliverables appearing in my mind’s eye as incoherent soups of platitudes and KPIs. Internal pressure to not push the tooling further to maintain operational continuity, squeezing against the mandate of an analyst to parametrize, to understand, to improve. The excitement of discovering significant solar inverter anomalies becoming a joyless, truthless chore divorced from reality.

I had 250MW of solar power at my fingertips - few asset managers will ever see as many. I had access to more solar capacity than ever installed in my entire country. People knew who I was, and what I could do. How could I be unhappy?

The Renewable Analytics Software Developer

“And when I opened my eyes, standing above me was the Creator.”

January 2023. There were more layoffs. The Head of Engineering informed me that I had been transferred to the brand new Renewable Analytics Software Development team.

Joining the first few team meetings was definitely humbling. My team consisted of the Head of Engineering, the CTO himself, a ranking manager, two staff-level developers, and two interns(the other one more experienced than me). Yet, this was exciting as well, as the doors have opened again and once again, I felt like anything was possible.

My first assignment was to automate a time-consuming weekly process, by refactoring part of my solar tooling and lifting it to run as a batch workflow on AWS. A staff developer was assigned to lead this project, and I was there to support. On our very first meeting, to my surprise and delight, he asked me to explain the architecture of the solar tooling in depth.

It was a simple request, but it genuinely meant a lot to me. Prior to that day, no one had bothered to understand the tooling, but only noticed when it was broken. And thus we had an extremely productive meeting, discussing architecture patterns and the appropriate cloud lifting strategy to use. We settled to let me lead the refactoring, while he would lead the lifting.

There were problems along the way of course. The biggest one was the beforementioned impedance between analyst-developed software and mentalities and production-quality software. One was built for quick prototyping and quick fixes, while the other was built to run flawlessly in a remote environment. Patrick, if you are reading this, I deeply regret not understanding this sooner and wasting your time debugging from the AWS console. I sincerely wish I knew better. Eventually though, after some consternation, we finished the project to wide approval.

“No. No. They manifested… like a flame. They weren’t really from anywhere. The conditions were right and they came into being.”

In the process, I became aware of the intricate web of microservices that the Clir platform is built on. Taking the time to map out each of their functions and capabilities in relation to each other was time consuming, but worth it. In my mind’s eye, I saw webs of nodes and interconnections. I saw threads of light streaming between fixed stars. I saw a leviathan of relentless machinery and software engineers scurrying around it. And I saw in my hand a spare, unwanted gear, the solar analytic tooling that I was so proud of.

Desperately, I wanted it to fit in, even though I knew it was the crude handiwork of an analyst in contrast to the precision-machined parts that were the other microservices. And yet, I knew that, given time, I was capable of overcoming its deficiencies. Poring through many single-line diagrams and software architectures, I concluded that the solar analytics tooling can remain relevant, not by emulating the rigour of the other microservices but by continuing to advance ahead of them as a flexible space to test new solar models, new analytic methodologies, and new product hypotheses. I drew these ideas up as a proposal, with appropriate arguments for their benefits, and sent them for review.

“You- You promised that we would change the world, together. You broke your promise…”

And the plans were rejected outright. Not out of malice or incompetence, but because blue sky thinking was no longer affordable to the company. Instead, the company wanted KPIs to be delivered to clients, more of them and with a shorter turnaround time. All new research on solar analytics, as it appeared, was to be shelved indefinitely.

For a time, I was resentful and depressed. But the fog of confusion was lifted, and I saw what high walls impeded the search for truth in a corporate setting. Striving for perfection in data analytics, in the eyes of management, is superfluous as long as we can read the clients a broadly correct and consistent story. Communication barriers between people, and even within people, keep people content and employed. I myself could not claim superiority - adjusting the parameters of my models led to a nontrivial spread of outcomes for clients.

The worst part was that I understood the rationale of the management to cancel further research. It would have been much easier to hate an arbitrary and discriminatory decision, rather than one made with sound reasoning by my respected colleagues and friends.

Nassim Nicolas Taleb, in ‘The Black Swan’, wrote something along the lines of “analysts who can’t abide to truth should consider not being analysts”. Father Zossima, in Fyodor Dostoevsky’s ‘The Brothers Karamazov’, wrote

Above all, don’t lie to yourself. The man who lies to himself and listens to his own lie comes to such a pass that he cannot distinguish the truth within him, or around him, and so loses all respect for himself and for others. And having no respect he ceases to love

I seriously considered resigning. But for unclear reasons, I did not. Perhaps it was cowardice and comfort-seeking, though I hope it was not. Perhaps I saw a filial responsibility to retain a good job, or recognized a social responsibility to give to humanitarian aid from a monthly salary. Maybe I felt that the search for truth has become intractable in any corporate setting outside this one. A part of me believed that existing difficulties can be surmounted by accumulating experience, even if experience comes from sometimes being wrong to clients. And finally, I knew that a search for perfection can be taken inwards, through reflection and study.

“The thing about perfection is that it is unknowable, it’s impossible, but its also right in front of us, all the time”

Anyways, I did my job well and improved my technical expertise outside of work through personal projects, reading textbooks, and earning AWS certifications. With the support of my friends, managers, and colleagues, I got into the rhythm of writing production-quality code and tests, and was quickly given the responsibility to design my own scripts, subsystems, user interfaces, and microservices. I continued to take the initiative to improve upon the solar analytics tooling for my old team of solar analysts, while accumulating knowledge about the subtleties of wind energy. I built a dashboard to visualize power curve changes in wind turbines, a tooling system to generate market insight reports for a new product line, a serverless automation to analyze wind turbine SCADA anomalies, new anomaly detectors, and new data quality checks.

I was at home, forging the links between renewable energy analytics and software engineering. The possibility-expanding power of analytics tooling and the relentless machinery of software architecture became complementary languages to articulate ideas by. But if technical expertise were languages, then there were many more languages that I had little awareness of. These languages were not spoken in repositories or pull requests or technical specifications, but behind closed doors and within implications and between silences. They articulate ideas not by snippets of code but through lines of credit and threads of influence.

Conclusion

Not a promotion

The morning after I completed a high-priority project, I was invited into the CEO’s office for a chat. I expected the reason to be a promotion, but the possibility of the other reason also crossed my mind. As it turned out, I did not get promoted that day.

Many, many software engineers, analysts and managers, all of them more experienced than me, had been laid off in the months prior - retaining me for much longer could have been unjust. Honour compels me to feel outraged at my dismissal, but I could not maintain it for long. In the following days, I had the opportunity to converse freely and candidly with many of my friends and colleagues. I was struck by a peculiar understanding, that of a chain of consequence rippling across every magnitude of power.

Humbly, I kiss the earth…

How many engineering contractors, equipment manufacturers, and asset managers have been positively or adversely impacted by the deliverables I helped to create and promote? How much have I built up or weakened the careers of my friends and colleagues? When an executive creates KPIs to retain or layoff analysts, the way we analysts create KPIs to grade renewable energy assets and their maintainers, do we recognize the ends of our actions? When Clir’s investors alternate between demanding rapid growth and quick profitability, when anyone puts money into an ETF or a mutual fund, or when analysts at a central bank recommend interest rates, can they hear the walls groaning, sliding under the forces that they bring to bear?

We at Clir want to exist above of this chain of consequence, cooly and impartially modulating the flow of information across different stakeholders for universal benefit. And yet, we are being constricted by the very same chain, and at the same time we may be binding others with it. If a similar dynamic plays out across the entire renewable energy industry, how can we be proud of the righteousness of our profession when we too, “wring bread from the sweat of other men’s faces”? If there is no escape from the bonds of dependent origination, if attaining impartial, absolute truth from data is a distant ideal, how can we approach our work today with integrity?

Is predation and calculated ruthlessness the rational recourse, disruption and corruption the tactical advantage, and desperate competition the natural order? Or is there a way forward in offering onions to the damned, in watering the earth with tears, and in perceiving a binding chain as links in an ecology?

I’m thankful for all my experiences at Clir Renewables, and I can only give their former and present staff my best wishes for their health and success. All things considered, Clir was an excellent place for me to contribute to renewable energy development, to learn more about software engineering, and to meet many talented colleagues and friends. It is a great company and I sincerely hope that they will reach their goals someday.

…but look to the stars

But there is still so much more to learn. And now, there is even more to do.