I was the person who first deployed Python at Goldman Sachs. At the time it was an "unapproved technology" but the partner in charge of my division called me and said (This is literally word for word because partners didn't call me every day so I remember)
Err....hi Sean, It's Armen. Uhh.... So I heard you like python.... well if someone was to uhh.... install python on the train... they probably wouldn't be fired. Ok bye.
"Installing python on the train" meant pushing it out to all the computers globally in the securities division that did risk and pricing. Within 30mins every computer in goldman sachs's securities division had python and I was the guy responsible for keeping the canonical python distribution up to date with the right set of modules etc on Linux, Solaris and Windows.
Because it was unapproved technology I had a stream of people from technology coming to my desk to say I shouldn't have done it. I redirected them to Armen (who was a very important dude that everyone was frightened of).
The core engineers from GS went on to build the Athena system at JP and the Quartz platform at BAML.
Armen was very passionate about the value of "strats" (GS's own term for "quants", and later broadened to include software engineers and data scientists).
A favorite quip of his:
At GS, I'm like an arms dealer. When a desk has a problem, I send in the strats, and they blow away all the competition!
Also, SecDB's core idea is not just tight integration between the backend and development environment, but that all objects ("Security" in SecDB lingo) were functionally reactive.
For example, you would write a pure function that defines:
When the market inputs changed, Price(Security) would automatically update (the framework handled the fiddly bits of caching intermediate values for you, so even an expensive Price function is not problematic).
This is loosely the same idea that drives React, ObservableHQ, Kafka, and other event-streaming architectures, but I first encountered this ~15 years ago at a bank.
It's as old as VisiCalc, it's how spreadsheets work.
I built a similarly reactive system for web UI binding back in 2004, running binding expressions on the back end with cached UI state to compute the minimal update to return to the front end, in the form of attributes to set on named elements.
This immediately struck me when I was reading this article.
To be honest, this whole paradigm seems absurdly fucking efficient for the developers. But I wonder about stuff like
* What happens if the data model needs to change? If you need to move something from db["some/path"]?
* How is it coordinated at a larger scale, how does everyone know what is running and how it interacts with everything else - can you figure out what depends on an object? What if the data used by your Price(Security) object changes and breaks it?
You write conversions and there's a registry where you register them to be picked up by the unpickler. If necessary you can also customize the logic that determines which version a given pickled datum uses to deserialize. There aren't so many guardrails when you're writing that stuff, but the infrastructure does its best to support you.
> If you need to move something from db["some/path"]?
There's support for both symlinks (db["some/path"] -> db["other/path"]) and for a kind of hardlink by making both paths point to the same inode-line id. You can usually find a way to do what you need to.
> How is it coordinated at a larger scale, how does everyone know what is running and how it interacts with everything else - can you figure out what depends on an object? What if the data used by your Price(Security) object changes and breaks it?
There's a common model for the things that are shared, and that has a versioning and release/deprecation cycle. Otherwise every type has an owner and you probably had to request their permissions to read their data, so you should have a channel of communication with them. But yeah people do rely on the fundamental business entities not changing too quickly, and things do break when changes are made.
True but not really helpful for this problem, because it can only tell you about the job you're debugging, whereas what you want to know is what code might ever depend on that data.
> This is loosely the same idea that drives React, ObservableHQ, Kafka, and other event-streaming architectures, but I first encountered this ~15 years ago at a bank.
See also the "observer pattern" [0]. It's a fun exercise to implement a reactive system in Python using the descriptor protocol [1]. IPython's traitlets library is an example of this in the wild [2].
Thank you so much for this context! When I started at Lehman in 2007 and for years through the i-banks & hedge-funds, 'Sec-db' was the North Star for so much me and my friends built. It's amazing to hear from the folk who brought that to life!
Took many more years for us to understand that we had learned more about banking and making money than our masters had learned about the potential uses of the platforms we had built.
We started Sandbox Banking. Many of our friends are at hedge-funds :-(. What's the career paths of those that first built sec-db?
I worked on Quartz for 3 years and loved it. Some devs grumbled about various aspects of it, but I come from an application support background and taught myself python, so I suppose I had fewer developer habits to un-learn.
From what I understand, all this started with SecDB at Goldman, which was a the prototype for all these systems but wasn't Python based. The lore is that SecDB was instrumental in Goldman being able to rapidly figure out what their exposure was during the 2008 crisis.
Some of that team, lead by Kirat Singh went on to start Athena at JP Morgan and then Quartz. I met Kirat once, he was considered a bit of a rock star in the bank tech community. He now runs Beacon, which is basically Bank Python as a service.
I work at Beacon.io and it's an awesome place to be. Kirat is indeed a rockstar and it's awesome to work with an CEO that knows great code. We also landed a Series C last month and we're growing :)
As a full stack quant dev who is struggling to go all-in on Beacon: Do people generally like to use Glint for complex applications? Are benefits of Beacon lost when interfacing from e.g. Angular? I'm afraid that professional frontend devs might be unwilling to work with a proprietary framework, but that's speculation from my side.
Glint is an integrated framework with the platform but it does not limit you to just using the framework. The platform is designed to be as extensible as possible, worry not about being locked in :)
I took a quick look, it seems like all the postings are London or New York. What's the feeling internally about remote hires? I'm assuming that's still out of fashion in finance and Beacon feels the same?
I know Kirat really well. Fun fact, one two-week dev cycle we had 667 distinct developers commit to the secdb code base which Kirat's boss described to me as "The number of the beast.... plus Kirat"
Second fun thing. Kirat was advocating for lisp for secdb for a long time and used to rag on me for liking python when it's so slow.
Interesting to be reading all this about SecDB. About 15 years ago I was offered a job working on SecDB (I forget exactly what the position was now). It and Slang sounded really interesting.
I do sometimes regret not taking the job because the people there were wickedly sharp and the tech sounded great, but in hindsight I'm not sure I would have thrived in a bank long term. I did a 3 month internship at Lehman's which I enjoyed, but I don't think I'd have suited a career in it. One thing I did get out of it was a total lack of fear around job interviews, if I could survive the 14 hours of interviews at GS and come out with an offer, then I can handle pretty much any recuitment process :)
It's amazing how a few people have left such a big mark on a part of the investment banking industry. I missed Kirat right before exiting BAML but met all his "disciples" and Dubno..including his miniature dinosaur and telescope in his office :). Very much felt like tech religion where no open debate on merits and drawbacks could be discussed. And a lot has changed in terms of engineering innovation with turnover since that era...
Why is the number of people who "left a big mark" so small?
1. An organization/industry can adopt each new technology only once. New technologies arise infrequently. Each time they arise, only a few people get to work on the projects introducing them. In other words, opportunities to leave a big mark are limited.
2. Credit for innovation is political capital. People hoard political capital and become powerful. They act as gatekeepers of innovation and take the credit for successful projects.
> From what I understand, all this started with SecDB at Goldman, which was a the prototype for all these systems but wasn't Python based.
Correct. SecDB has its own language, S-lang, which you could probably best describe as a "lispy Smalltalk, with a hefty sprinkling of Frankenstein". The concept of SecDB was so good that other large banks wanted to get their own. Athena and Quartz have been mentioned several times in this thread, by people far more knowledgeable than I could ever be.
It's not just banking, I know of at least one large pension/insurance company who are building their own version of SecDB, with direct lines to GS. (They don't try to hide it, btw: the company is Rothesay Life.) The last time I talked with them, they were looking for Rust devs.
> From what I understand, all this started with SecDB at Goldman, which was a the prototype for all these systems but wasn't Python based. The lore is that SecDB was instrumental in Goldman being able to rapidly figure out what their exposure was during the 2008 crisis.
Correct. We used python for a bunch of infrastructure stuff (eg distributing all of secdb to all of the places it needed to go). The actual pricing and risk was written in Slang with a lot of guis that were "written" in slang but actually that caused autogeneration of JIT bytecode that was executed by a JVM. Most of the heavy lifting behind the scenes was C++. So a bit of everything.
My grandpappy always told me to cut out the middleman.
Modern C++ was heavily influenced by the need to make it simple to use directly. If you are in the business of writing code instead of reminiscing, you can now leverage move semantics, lambdas, and smarter pointers to create software that is close to the silicon.
Python might be great, but it sure is slow. Its success is founded on smart people making it easier for not so smart people to call C++ that does the heavy lifting.
A big force multiplier in the old GS secdb model was simply the speed of the dev cycle vs speed of the code. As a strat you could push slang changes to pricing and risk literally in minutes with full testing, backout, object audit logging etc.
C++ changes went out in a 2 week release cycle so changes were still fast by most standards but much slower. But yeah we had 20m + lines of C++ code so it was extensively used.
For some context (as an ex-Goldman employee myself), "Armen" in the quote is most probably https://www.goldmansachs.com/insights/outlook/bios/armen-ava... , who has quite a legendary reputation within the firm for the work he's done. He was also one of the first to be hired as a "strat", which used to be how Goldman referred to its quants who sat between front office and tech systems and worked with both sides.
SecDB/Slang originated around 1992 at the commodity trading shop J Aron, which GS had bought 1981. Later (end of the 90s) there was a push to extend it to the rest of the firm, first fixed income, then equities. Armen flew the whole world-wide strat team to NY and gave a presentation, and to drive the point home, he played a clip from StarTrek: "You will be assimilated. Resistance is futile!"
I worked on Athena at JPMorgan for 8 years, and loved it.
Seeing Python at the core of trading, risk and post-trade processing for Commodities, FX, Credit etc was such a great developer experience.
By the time I left JPM, there were 4500 devs globally making 20k commits weekly into the Athena codebase. (I did a PyData presentation on this [1] for more details).
The one downside was the delayed transition from Py2.7 to 3; I left just as that was getting underway.
There are a number of workplaces where I'd have been willing to rely on "probably wouldn't be fired", but a bank is definitely not one of them. Congratulations on shipping something useful in the face of that risk and uncertainty.
The trading community walk on a knife edge all the time, it's not a place for the faint of heart. I used to support derivatives trading systems and a few times there were issues that meant they'd lost control of orders on an exchange. Scary stuff. It requires a crazy mixture of careful, deliberate, calculated risk control on the one hand; but once you commit to something you jump in with both feet and throw everything into it.
You need to be both meticulously risk averse, and also willing to do whatever needs to be done when it needs doing, and accept responsibility. It was great!
I worked at UBS on the G10 FX options desks in Stamford and Singapore. Remember being very surprised by how my interview and training were incredibly stressful. It was very intentional as well, where my trainer knew the exact reactions he was creating with his behavior.
It was only a couple of weeks in, when I had to react within 60 seconds (USDJPY option expiry, NYC cut) on a position our front book would have lost MM on; with senior sales MDs screaming at me as well. Lo and behold, I was just used to it and could focus and execute based on my training.
My wife calls my thinking on the training - Stockholm syndrome. I still believe those skills were incredibly valuable for me, just perhaps delivered in a more 2021 acceptable approach.
Investment banks are basically risk-management shops. The partner made an assessment and evaluated the potential benefits as higher than risks. Note the word "probably".
Almost definitely false. Provided you weren't doing anything intentionally malicious with it, the risk would be that regulators might fine the bank for inadequate controls. As such, the bank might fire you for doing something that could lead to such a situation, but I don't see a criminal charge. There was actually quite a decent bit of "unapproved" software in use at one of the banks I worked in - mostly stuff that was in the process for approval, but that could take forever, so it was reasonably common for teams to run through the checks themselves (security scan, license review, etc) and move forward while the official review confirmed no issues.
Well the login message I was greeted with on every ssh connection certainly threatened criminal prosecution for unapproved software at the extremely large bank I worked at.
Unlikely? Sure. But a lawyer somewhere thought it was worth reminding me 10x/day, so going to assume it's possible provided your unauthorized software caused a serious monetary loss.
I joined the BAML grad scheme 10 or so years ago. We had a presentation from one of the Quartz guys and someone asked how they’d manage upgrading the version of Python. They were using something like 2.6.5. The whole move to 3.x was a thing. The Quartz guy just flat out said they wouldn’t upgrade.
Seemed crazy to a new grad back then, but now I wouldn’t want to consider it either.
Thanks for your contribution! It was amazing that even in my role where I didn’t use Quartz, I could see and search all the code. Felt quite novel back then.
A friend I've known since I first started on Wall Street now rides herd on the BofA Quartz libraries. One component of his job is to make developers aware of existing libraries they can use to solve business problems instead of reinventing the wheel. His theory of why they always have excuses not to do that is that they have no training in software development. They are still at the point in their learning where the are just excited that they can press keys on a computer and get it to do things they are barely able to understand.
I remember Slang when I first saw the code, a parse tree based evaluator in 1997. Come on folks. Separate parsing from evaluation. Opaque types with message passing. Inference anyone? Clearly no one read hindly milner.
Add parse time optimizations, add locals, hey globals and locals weren't handled properly. Python in the 90s anyone?
Shitty KV store with 32K object limits related to some random OS2 btree limits. Add huge pages.
Deal with random rubbish related to inconsistent non transactional indices.
Figure out you should layer nodes in a dag. A dag is topologically 2 dimensions, fairly limiting.
Figure out that's somewhat similar to layering databases, it's just another dimension.
Hmm bottom up evaluators, you actually need top down as well, create a turing complete specification. well, limit it a bit.
Ah KLL points out that layers on top of dimensions end up being combinatorial, but you can actually cache the meta traversal and it's small n.
Lots of people point out category theory parallels. Haskell is pretty but completely unusable. I'm a math guy, and it's still unusable, I don't like feeling smart when I do simple things.
But interestingly creating imperative functions with pure input/outputs with implied function calls is pretty interesting. You can create an OOP paradigm with self and args as known roots aka linda tuple spaces.
Ah and each tuple space point can be scheduled independently, some issues with serialization and locality...
Go to another bank and choose to use python, foolishly decide to rely on pickle. Do that twice. Bad idea.
But write a much better petabyte scale multi-master optimistic locking database with 4 operations. Insert, Upsert, Update, delete. WAL as a first class API.
Finally decide that writing a coding scheme to convert python objects to json is not really hard. And of course cloud native and cloud agnostic is really the only way to go nowadays.
I'm always confused why people complain about Athena/Quartz, hell we wrote it all, fix it if you don't like it. Open source it if you want other people to contribute. If we made stupid decisions on pickling data, well there's a version id, add json serialization, it's not hard, don't take things as given.
Oh wow, I remember Quartz at BAML… Though this was several years after initial deployment and when core devs left.
One day I will sit down and write a small poem about the insanity of software development based on my experience with Quartz. It will be an intriguing story of love and hate being told through a sensual dance between sales and engineering. The battles will not be epic but the consequences of one’s actions will be far reaching.
Tell me more about what worked and didn't..I recall the pain of watching QzDesktop load and Bob/HUGS jobs failing..but what else, and what did you enjoy
When all you have is a DAG all solutions end up with sales person coming at your door problem. But years down the line we didn’t have traveling salesman problem but out of date libraries problem.
The monkeys tried to make their risk taking more safe by introducing all kinda of random constraints not realising that what gave power to the whole idea was actually risk taking.
And no, they didn’t try to manage risk but have completely fail to understand that risk is what made product good in the first place.
And this is how every great idea in human history fell apart, the original people had different view of a current problem however the following generations have only understood the simplified problem and down the line the the solution to original problem had became the actual problem.
I still believe the poem would make greater justice to the whole idea, but what I tried to explain here is that the simple things we have all witnessed actually hide much deeper truths about life in general.
And from there I assume QzDesktop loading times where not the issue at the beginning of this deterministic chaotic system, but simply one of the possible generational products.
I am yet to understand how to solve the monkey issue.
Curious why would this high-powered person go to bat for a technology decision they didn’t seem to have done any risk assessment of? Wouldn’t he be liable if something was exploited and hurt and company, like his head would be on the chopping block for giving the go ahead when they traced it up the chain of command?
This is only my opinion, but I think the reason Armen said it like he did was because by not making it an order he's giving Sean the option of not doing it, if he's not up for accepting the risk. However the risk was both of them could have got fired.
Armen must have known people would know Python had been put on these machines and that he authorised it, in fact what's the point of putting it on them if nobody knows and nobody uses it? I can guarantee you that within 24 hours someone was asking Armen why he'd authorised this and was justifying it. There cannot have been any possibility of dodging responsibility for this decision. If anyone got fired it would have been Armen, with a possibility of Sean going as collateral damage.
This is the big league. You make your decisions and you accept responsibility for them.
Exactly, so what is this Armen character getting out of this other than a potentially big amount of liability and unarticulated risk.
The OP said he told people openly that Armen told him he could do it when asked.
This makes no sense to me, what’s the upside to Armen? If he is business savvy, he needs to be gaining something in exchange for having his name thrown around by OP as signing off on this.
>In 2011 Goldman Sachs put its top computer wizard, Armen Avanessians, in charge of the division. He has helped turn round its fortunes. The arm’s assets under management reached a nadir of $38bn in 2012, but it now manages $91.8bn...
Maybe that bank is more generous. The one I worked at begrudgingly counted out the pennies like it was coming out of the war orphans fund or something.
He's doing his job, which is to ensure people have the tools and resources available to do their jobs. You know, furthering the goals of the organisation.
You say he reached out to you and asked if you liked Python. He probably wanted to roll out Python and was looking for someone who wanted to do it. If he told someone to do something they weren't passionate about, they would fail. He wanted to make sure it succeeded, so he reached out to you.
If he's such a bigshot and everyone was frightened of him, he must not have been afraid of them. When he said, "you wouldn't get fired," he probably meant what he said. He was giving you air cover. And it worked. When the gnomes came out after you, you just sent them to him. And they didn't bother you again.
I can imagine how the conversation went:
"Armen, did you tell Sean to install Python"
"No, did he"?
"Yes he did!"
"Great!!"
Now the gnomes are on their backfoot and have to defend why Sean shouldn't install Python If this guy Armen told you to do it, Armen has to defend himself to them.
Hehe. Well I guess you would have a unique insight into his thought process. ;-) But yes indeed that's certainly another explanation and it did indeed work that way.
Well thanks for the air cover, and for all the other opportunities you provided for me and others at GS. I really appreciate it. It was an amazing time and I learned a great deal.
I worked for GSAM for a bit as my first NAPA project - I guess you're referring to Armen Avanessians? Haha haven't heard references to the "train" in so long. Did you ever do any Slang/SecDB dev? I was mostly in FICC Tech so was pretty much slangin' slang most of my time there.
JSI (Java Slang Integration) was just getting off the ground but there wasn't too much for the front office tech teams to do there until it was to mature in the coming years.
Yes, that's who I was referring to. I did absolutely tons of slang and a fair amount of TSecdb as well as a lot of work on the C++ infra and the build and distribution code.
There was no single person that introduced python at Citigroup that I am aware of. It came in via a variety of teams mostly because of the fact that the alternative was perl, and no one wanted to write perl (yet somehow kdb was acceptable a few years later).
Because it was unapproved technology I had a stream of people from technology coming to my desk to say I shouldn't have done it. I redirected them to Armen (who was a very important dude that everyone was frightened of).
The core engineers from GS went on to build the Athena system at JP and the Quartz platform at BAML.
//Edit for grammar