Every single place that I ever worked at in a past 20 years tests payments using real cards and real API endpoints. Yes, refunds cost a few pennies and sometimes can't be automated, but most payment providers simply do not offer testing APIs of a sufficient quality.
Situations when a testing endpoint has one set of bugs not found on production and vice versa used to be so ubiquitous in mid-2000 to mid-2010s, that many teams make a choice agains using testing endpoints altogether - it's too much work to work around bugs unique to the environment that no real customers actually hit. And now the whole generation of developers grew in a world of bad testing APIs of PayPal, Authorize.net, BrainTree, BalancedPayments (remember them?), early Stripe, etc. So, now it became an institutional knowledge: "do not use testing endpoints for payments".
To be exact, people often start using testing endpoints for early stages of development when you don't have any payment code at all, but before the product launch things get switched to production endpoints and from that point on testing endpoints aren't used at all. Even for local development people usually use corporate cards if necessary.
I have a suspicion that things may be different in the US, with many payment providers' testing environments simulate a typical domestic US scenario: credit cards and not debit, no 3d-secure, no strict payment jurisdiction restrictions, etc.
I've worked on adding Google Pay / Apple Pay to the mobile app of a large European ecommerce company, and that's more of less how we went about it.
Start with the sandbox / test environments, once you get reasonable responses end to end, release the thing behind a feature flag. Backend moves slowly, so you add stuff to the mobile app that really belongs to the backend, but f it, because it's the only way you meet the deadline, you will convince someone on the backend later (aka never) that it's backend responsibility. Ask (pressure) the developers into buying some cheap stuff from your shop with their own credit cards, because the company is a behemoth and approving real credit cards for testing would just take ages and you want to release yesterday. It annoys you, but you realize that 5x5 euros is worth getting this done rather than start fighting a losing battle against company processes. Cancelation is possible, but it will take a couple of days. If there's any issue, you debug it across a bunch of teams and/or companies. Things start to work most of the time, time to release the stuff to x percent of your users. Check analytics and error logs frequently. Some production users got their payments through, increase rollout percentage. You discover more and more undocumented error codes, you improve the error messages to the users so that they don't retry 10 times with a card without sufficient funds. After a couple weeks, things start to stabilize, you move on to a new feature...
The test environments were so complicated and had so many caveats that whenever I had to do something, I had to re-read the docs and our notes to know all the "traps" we already discovered. For the people who didn't work on this payment feature from start to finish, testing in the officially recommended test channels were hopeless.
If the employee can get it refunded with an expense report, like any other work expense then in my experience (in California), it’s not illegal. I’ve made plenty of work expenditures with my personal CC that I get reimbursed from the company with an expense report. But *pressuring* employees to do it is plain wrong (and may be illegal).
Coming back here, not sure if anyone still reads this thread a day later...
I am not sure about the legality of it here in Germany, but I'm not really sure I could even prove it.
Pressuring us into buying these items are hard to prove, nobody said we need to do anything. You want to be someone who makes sure the feature can launch on time and works correctly, or you want to complain (rightfully) that using your own cards should not be necessary for testing a feature?
It was, though, implied 1. we need to make sure the product works and shipped on time, 2. you can't do it without using your own cards.
I know people on the team who simply didn't test, but as it was a feature I was mainly responsible for and genuinely interested in, I wanted the launch to be successful.
We also eventually got the money back (most of the money? didn't check them all).
In the end, it was in total about 25 euros, and that's not a sum that I would sue my employer over, especially as I was "happy enough" at the company.
>Here in the state of California, labor laws define that an employer cannot require a team member to take on expenses that are an integral part of the job.
Two comments. (From a non-CA legal jurisdiction context.)
You can always ask (but not pressure or require). Make sure you make it clear there's no downside to them refusing.
Another option I've used is to hand cash to co workers and ask them to spend it on their credit card for testing. I've rarely had anyone refuse that. (A few very junior staff members who were maybe right on the credit limit on their cards I suspect.)
California is an example, but I’m unclear about other states. Honestly, I’m unclear across the board because there are a lot of employee made purchases that are conditions of employment (phone, computer), and it could be argued that this purchase is a similar necessity especially if it’ll be refunded.
Any job that asks you to buy your own equipment, especially a computer, is a scam (unless you are a freelancer, in which case you should already have equipment)
We built against Stripe's sandbox and never had to test in production, so I never had to use a real credit card. It may have happened when going live for the first time, but that would have been two/three charges with one time payments and recurring payments (hardly what you'd call robust testing). Issues observed in production can usually be reproduced in the sandbox. There's some other caveats between the environments though that I'm forgetting but I don't think we ran into those.
We also had an ACH payment provider (add your bank account, verify it, we deduct from it, etc.) that also had a sandbox and had no issues there either.
> We built against Stripe's sandbox and never had to test in production, so I never had to use a real credit card.
Did it, got bit in the ass when some workflow was disabled in production and not in their sandbox. I don't recall the exact thing but always fun to push into production all sure of yourself to have to rollback fast and ponder why you get some fun message in your logs. At least the error message was clear about what we were doing being available only on the sandbox.
It was some years ago so I don't remember if it was in Stripe Connect or during the mandatory 2FA rollout.
Stripe is the best I've used, but it has a ton of issues to this day:
A) Account settings are specified separately in prod / staging
B) Differences between those environments are not automatically reported in a useful way
C) Only one staging env per customer. Want to check what a new setting will do? Every developer is getting that setting turned on.
We also built our Stripe integration using only the testing tools they provide. The first real payment in production was completed by our product guy, but only as a sanity check. Maybe we got lucky, or maybe Stripe is just that easy to integrate with, but we haven't been surprised by any production behavior.
A "sanity check" is a test. Alway always always test that real prod payments end up in the bank account they're supposed to before whoever can rollback goes home for the day. _Always._ And document the existence/process/details/outcome of that "sanity check" test.
Not documenting the existence/process/details/outcome is exactly the distinction that I intended to make when I called it a "sanity check". I was really just offering a data point, not a comprehensive guide to best practice.
Slightly off-topic, but has anybody else seen an issue with stripe's live checkout flow when you collect a phone number? People paying with Apple Pay will frequently have the first 2 or 3 digits doubled! So like 313105551212 will come through. Didn't seem to happen in the test environment.
(Oh also sometimes it defaults users (in the U.S.) to Anguilla as their country code (also +1) but then gives users an error their phone number is invalid.)
Same at my current and previous company - the Braintree and Stripe sandboxes worked fine for us, and post-release it was just a case of monitoring both integrations (and doing a test sale on prod if it was a quiet period).
> Situations when a testing endpoint has one set of bugs not found on production and vice versa used to be so ubiquitous in mid-2000 to mid-2010s
Honestly that's still the case, at least with Adyen. At $pastJob we had a pretty robust regression suite that'd regularly run into breaking changes in their test environment (against _old_ versioned api endpoints!). We seriously questioned whether or not anyone else used them since they were never aware until we submitted tickets. This also falls apart as soon as you use any of their "special" features that require an account rep to enable, they just don't work in the sandbox.
Another pain point is that the test payment methods offered are static. You can't setup cards for specific scenarios, e.g. tokenize, successful payment, then it expires -- you can only test an expired card as a one-off.
I just ran a couple hours ago into such a bug with Adyen, a couple payment methods have completely different behaviours in test vs production, and we ended up having to test in prod anyway.
Stripe considers using a real card to test a zero-tolerance fireable offence.
I, as the name on record for an organization with Stripe, made an actual, legitimate payment to said organization (and did not refund it). Stripe's automated system caught the payment and terminated my account automatically by the time I woke up the next morning; fortunately I was able to reach a real human and explain that in addition to working for the non-profit, I was also donating to it, and they begrudgingly restored the account.
My company has hundreds/thousands of ecommerce clients, of which a huge portion use Stripe and we do real card testing every time we deploy code that could affect payments and its never been an issue.
How do those places ever pass a PCI audit? One of the first things the auditor asks is "please show me proof that your testing is never done with real credit cards"
(Unless they're getting their test environments PCI certified, which sounds like a waste of money.)
>Every single place that I ever worked at in a past 20 years tests payments using real cards and real API endpoints. Yes, refunds cost a few pennies and sometimes can't be automated, but most payment providers simply do not offer testing APIs of a sufficient quality.
I think this means their own real-money credit cards, that they spend a few bucks on for testing purposes. Not customer credit-card data.
Makes sense. What are those volume limits like? I suspect the lion's share of companies like Stripe are those under those limits (since the largest companies are willing to trade simplicity for better rates)
edit: according to a quick search, it looks like it's 6M transactions/year to require an audit vs self-assessment
My last company, Rainforest QA, developed issuing virtual credit-card numbers for just this purpose - kinda like the ones used for privacy.com. Customers use them for testing in prod. Simple, effective - and testing the exact same flow as customers.
Before this, we found a lot of teams using either their own corp cards, or pre-paid visa type things. All, a pain to manage balance wise.
Seemingly, the biggest problem left with doing this is production metrics; these transactions in prod tend to affect the main metrics - either your own, or payment related things.
Is doing a smoke test in prod with corporate cards bad practice?
We are rolling out subscriptions with Stripe and an internal business unit is will actually be using the service so they put it on a company card. Basically they're our first live customer to test all the prod systems. No refunds or anything.
No, it is not bad practice. Only developers who don't care about money actually coming through the door -- the same ones that get caught up in a local maxima trying to perfect the imperfect -- say that it's bad and that you should not do it.
Regardless of what developers think, the payment providers generally forbid it. For example, Stripe says:
> Don’t use real card details. The Stripe Services Agreement prohibits testing in live mode using real payment method details. Use your test API keys and the card numbers below.
I have to imagine they’d only care if you were running significant volumes of test transactions and refunding thems, like if you were using live credentials in a dev environment.
Either way I’d be hard pressed to deploy significant changes to payment-related code in production without at the very least seeing a real $1 charge go through and everything work as expected. The risk of a ToS enforcement for this seems much lower than the risk of some bad logic in an if (env == ‘prd’) making customers unable to give you money.
This is my read on it as well, but I'd really like an official clarification.
Rare production smoke tests are in a gray area. They may be technical violations but they're allowed to happen as long as they stay infrequent and above-board (company card, small amount, no chargeback, etc.).
I think you're misunderstanding here; people are talking about a smoke "test" using a real credit card against the real production payment system, using production API keys/authentication/etc, with real money moving around. No payment provider forbids that.
I can't find any clarification on this by a search alone. When I went looking for it in the actual services agreement, I couldn't even find any clause about testing at all.
Among other things, you're not allowed to use your own credit card to make a purchase where the money will come back to you, because credit cards want to charge cash advance rates for that.
But it's not a purchase. You're not exchanging real goods or services for that money (unless your smoke tests run a lot deeper than mine). Your motives may be benign, but from a legal/regulatory perspective, it's a suspicious transaction.
Yes. By the letter of the agreement, you are not to use your cards to do test purchases against your account.
You ocassionally see complaints about payment processors when microbusinesses do this and get banned. So it is something that does get checked ocassionally. (There's a top level comment about this)
I think the payment processor doesn't want you to do it because you may issue many transactions and then refund them which incurs cost, or you may be using it for manufactured spend which incurs issuer ire. Maybe it's a brown M&M thing; if you didn't read that part of the agreement, you didn't read anything else, and they may as well kick you out early and avoid hassle.
Generally speaking, no one is getting banned from Stripe for the occasional transaction tested in production, come on. If this was the case, virtually every company I've ever worked at would be banned from Stripe. It's reasonable to confirm your system actually works once deployed outside of test environments.
No disagreement from me that is what the letter of the Stripe service agreement says, but what happens in reality is clearly different. I take that rule as trying to encourage people to use the very good test environments Stripe offer, or to limit scale of test transactions in production, rather than trying to shutdown a paying user (the company) for trying a legitimate transaction in prod with a legitimate card. I have no idea why you would want to risk the first ever transaction in prod being performed by a real customer - why leave it to chance that it is not setup correctly?
I have also been on calls with Stripe support staff where we tried a card transaction in production for testing purposes, FWIW.
Also very interested. I wonder if we could get a comment from a Stripe person or a recently ex (like @patio11 ) to clarify what’s allowed and what’s just ignored.
This is also true for cryptocurrency. There's always a testnet, you use it whenever possible cause xact fees are high, but it never works the same as real.
I built software for the self-service automats of a very large European rail company, and I remember spending whole afternoons testing every possible scenario listed in our testing book.
Those required manually purchasing tickets to many different cities, using the credit card terminal on the machine. They even had fake Discover, VISA, Mastercards you were to use to make those test purchases (to this day I don't know who came up with those fake cards and how that part of the system worked).
But no, not everybody tests payment in production. It was a while ago, but I don't think they'd have switched the entire philosophy by that much since then.
PA DSS/PCI DSS standards say you aren’t supposed to test with real cards. It’s a certification criteria. I had to make my own
PCI DSS (Payment Card Industry Data Security Standard)
Requirement 6.4.3 of the PCI DSS states:
"Production data (live PANs) are not used for testing or development."
This requirement is aimed at ensuring that real cardholder data (Primary Account Numbers or PANs) is not used in non-production environments, such as testing or development environments, to minimize the risk of exposure and unauthorized access.
PA DSS (Payment Application Data Security Standard)
Requirement 6.3.4 of the PA DSS states:
"Production data (real PANs, track data, or other real cardholder data) is not used for testing or development."
From any engineering post I have seen about Uber it seems like some deceptive marketing / hiring tactic when it is analyzed . They seem to repurpose standard practices into a blog post (this one might sounds like it has been derived from one).
I seem to remember about 4111 1111 1111 1111 times that I tested a payment system with a card that wasn’t real, although I acknowledge that when I was done convincing myself that I was a good programmer, I would almost always be disavowed of this notion after using a real number.
Eh, no? I've never tested payment code using real payments. Ever. The idea of doing it with real payments is pretty out there in my book even :)
Then again, every payment provider/bank I've integrated with, had decent testing end-points and we often even support them in production. i.e, you can select a staging/testing env of you provider to test order flow or whatever.
Nooope! Services like stripe let you test the payment workflow in their test environment which works exactly like the production one but without the payments going outside Stripe.
Every single place that I ever worked at in a past 20 years tests payments using real cards and real API endpoints. Yes, refunds cost a few pennies and sometimes can't be automated, but most payment providers simply do not offer testing APIs of a sufficient quality.
Situations when a testing endpoint has one set of bugs not found on production and vice versa used to be so ubiquitous in mid-2000 to mid-2010s, that many teams make a choice agains using testing endpoints altogether - it's too much work to work around bugs unique to the environment that no real customers actually hit. And now the whole generation of developers grew in a world of bad testing APIs of PayPal, Authorize.net, BrainTree, BalancedPayments (remember them?), early Stripe, etc. So, now it became an institutional knowledge: "do not use testing endpoints for payments".
To be exact, people often start using testing endpoints for early stages of development when you don't have any payment code at all, but before the product launch things get switched to production endpoints and from that point on testing endpoints aren't used at all. Even for local development people usually use corporate cards if necessary.
I have a suspicion that things may be different in the US, with many payment providers' testing environments simulate a typical domestic US scenario: credit cards and not debit, no 3d-secure, no strict payment jurisdiction restrictions, etc.