Chris Fry, of Salesforce.com recently posted a long blog entry entitled, “Why over 40% of Salesforce.com’s traffic is Web Services.” It’s an interesting read, and really a decent marketing piece for those technically oriented. Let me give another view.
One of my SFDC clients makes anywhere from 30,000 - 100,000+ queries to the API per day in just one of his integrations. In fact, if noone touches any records he’ll make over 27,000 queries to the API on any given day. If we add in the number of subsequent query(), retrieve(), create() or update() calls we’re looking at anywhere from 50,000 - 200,000+ calls a day. (Depends on how busy people are.)
Yes, this client is likely responsible for up to 2% of the total API calls (from the figures in the blog above), yet represents 0.0025% of SFDC’s subscriber count (from their homepage). Here’s the question I’d like to ask:
Why do I think that 95% of those API calls shouldn’t have to be made? (I’d have gone with 40% for effect, but frankly, it’s really more like 95%…)
- Joins in API queries. Take your typical invoice data structure. A header object, a detail object, and a payments object. Lets toss in the SFDC standard notes and activities for good measure. To get all that data for a given invoice from the API you need to make a call for the invoice, and 1 call for each related record, or 1+5 calls. 6 calls for what should be 1. An >83% waste for this example. Do it with Opportunities (include the account, contacts in the roles, line items and products for good measure and you’ll see where it gets nasty).
- API as a feature stopgap. Lets take our invoice example above. We need to keep our invoice totals nice and fresh for our users. So we use formula fields on the items to keep the line item total accurate, but how do we keep the total amount in the header accurate? We use a formula field. Uh, wait, formulas don’t span relations. Uh oh, here comes the API calls. When we convert leads we need some data copied to buth the account (or contact) and the opportunity. Can’t do that either. Here comes the API again. The list goes on and on…
- To top off our top three here. I don’t know when users edit records. I tried coding a digital Ouiji board to figure it out, but I still had to ask the API every-so-often to see if there were any updated records. One call for each object. So if I’m monitoring Accounts, Contacts, Leads, Opportunities, Products, Assets, Invoices (and it’s 2 important children), Contracts, and a custom object and I’m at 11 calls just to see what’s new. Do that often enough so your users aren’t waiting around for whaterver data your API calls are going to update and wham, you’re pushing thousands of calls a day. Get the API to make outbound calls on record create/update/delete.
Here’s my forecast: If SFDC were to implement just these 3 features, and all the API tools/integrations were to start making reasonable use of them, I think the number of API calls a day could drop from 10 million/day to 1 million/day. *(Yes, I know not every developer will switch over right away, and not everyone runs optimal code, but that’s not my point.)
Oh, and to the SFDC marketing department — if that actually happens, it’s a good thing.
License
This work is published under a Creative Commons Attribution-ShareAlike 2.5 License.
I agree. The major usage of the API isn’t because it’s so wonderfull and we all love it to death. It’s because people are forced to use the API to make up for major issues with SFDC’s design.
That being said, the API is a very nice thing to have, I just wish I wasn’t forced to use it to cover for lack of features in the core product.
Comment by E.J. — July 28, 2006 @ 7:50 pm
The Heretic’s View on Salesforce.com API Traffic…
Wide awake after a 60 day hibernation, the Salesforce Heretic has a nice piece on Salesforce.com API traffic. A post today from Salesforce’s Chris Fry appears to have been the spark
Related Posts:Access Salesforce.com Reports ProgramaticallyBlog t…
Trackback by SalesForceWatch.com — July 28, 2006 @ 9:39 pm
And don’t forget lack of caching, upsert, and compression as poor usages of the API.
How many times are describeGlobal and describeSObject unneccessarily called instead of using a cache?
upsert is another great API method that rarely gets used.
And every call to the API from .NET apps, by default, are not compressed, adding megabytes of bandwidth on top of the excessive API calls.
But hey…. SFDC is running the meter on these calls and has the capability to bill for any excess. These are all consumer problems… not provider.
Comment by Mike L. — July 29, 2006 @ 12:57 am
Mike L, let me take your comments point by point…
Caching is a double edged-sword. If you cache too long you’re using old data, too short and you add load. What is an acceptable cache expiration period for one org is likely different for another.
Upsert calls are also nice, if the things you want to do with the API can use them in the first place. Also, keep in mind if you write AppXchange apps, you can’t use upsert unless you create the field manually. Lets chalk that up to another incomplete feature.
Compression… Again arguments both pro and con. If you have an excess of bandwidth available, and are CPU contrained not using compression on calls benefits everyone by not adding the load of the compression/decompression on the API call. Given the instablity early in the year, I disabled compression on my calls to try to help shed load. Also keep in mind that compression doesn’t mean you make less calls, just that your calls are smaller in bytes, but harder on the CPU.
Lastly you refer to SFDC “running the meter” on the calls. They know (probibly better then I as I don’t normally aggregate those stats unless running in debug mode) exactly how many API calls this client makes in a given period — and I’ve spoken to them about it. At one point they were in the top 5 across all customers and we had a discussion at length as to why, including exact use cases.
Simply put - if what is billed as the world’s most customizeable on-demand CRM application requires me to issue an inordinate amount of API calls to stopgap missing functionality, that’s not a consumer problem. (And Summer ‘06 *cough* did such a great job of bridging that gap — but that’s another discussion entirely)
Comment by heretic — July 29, 2006 @ 2:12 am
I’m not a liberty to say much (yet), but I’ll make an offer to ya’ll - I’ll be happy to host this debate in person in San Francisco, and provide the beer.
Take a look at http://www.salesforce.com/conference/developers.jsp (and if there is something missing you’d like to see let me know.)
And since is the Heretic’s blog, I’ll let you know that if you register as a developer you can use the AppExchange Seminar code (DEVGR06) to get an additional $100 off.
Comment by Adam G — July 29, 2006 @ 12:45 pm
Let’s be realistic on two topics. First - the AppExchnage is less than a year old and the fact that over 300 companies have build applications - granted mostly trivial ones - means that the API is reasonably functional and we can only expect it to mature.
The second and most important point - never mistake the salesforce.com API for a developer library like you are used to have from a java package or a C++ lib - the API will always contain only as much facilities as is useful for Salesforce.com itself to grow salesforce.com ’s customer base - and not yours. Salesforce.com is in the hosting business, not in the software development business.
Comment by Dejan Nenov — July 29, 2006 @ 3:05 pm
I don’t think anyone is saying the API isn’t functional. What some of us are saying is that there are several changes that could be made to make things much more functional for us AND that we shouldn’t have to use the API as much as we do in the first place.
Some changes that would drastically improve the situation:
1) GetUpdated() call that will work on multiple objects at once, all objects in the system and allows fine grained time parameters instead of being truncated to the minute.
2) SFDC-side event triggers for data changes (all CRUD events) that can fire a client-side web service. No, the current workflow and e-mail notification is not acceptable for this. I would guess that a VERY large percentage of API calls are just people polling for data changes because there’s no better way.
3) A better workflow engine built-in that allows actual data changes based on other data. Instead of just being able to send an e-mail or create a task, it should be able to set values, create objects, etc.. (perhaps execute an scontrol?)
4) User schema versioning so that it’s easier to cache the SFDC schema for a particular user. Something simple like a version number that is returned when you log into the system and is incremented every time a configuration change is made that affects that user (new fields, fields removed, new objects, etc…)
Comment by E.J. — July 29, 2006 @ 3:43 pm
Dejan,
“Salesforce.com is in the hosting business, not in the software development business.”
Your comment really brings me to a core problem @ SFDC…
They are a CRM provider. They are in the hosting business (with the AppXchange edition — no CRM). They are catering to large companies, small companies, not-for-profits. Companies in the B2B space, B2C space, and probibly a company just in space…
They have companies with large integration requirements, no inegration requirements, extensive processes, and simple processes.
What unites them all - SFDC wants them all as customers.
You can’t be everything to everybody.
Comment by heretic — July 29, 2006 @ 6:35 pm
Why Building Scalable Web Sites is only Half The Problem…
Creating Scalable Websites
One concept I try and use in understanding the new world of feeds, open APIs and brittle but p…
Trackback by James Governor's MonkChips — July 31, 2006 @ 9:09 am
I suspect the asynch callback query feature, if announced at Dreamforce, will remove the need for constant pounding on the API and polling for changes.
As for JOINS, I don’t know. This is on my wish list too. I’ve developed the (bad?) habit of creating a lot of intersection objects, but this really starts chewing up the allotment of custom objects.
SFDC is doing a pretty good job of providing familiar, old school relational concepts and constructs, although their underlying metabase schema is clearly not condusive to adhoc queries.
The SOQL to physical SQL translation layer must be a nightmare to refactor, refine and tune.
Has anyone tried building star schemas and OLAP cubes from their SFDC data? That would be the BI approach to this problem.
Comment by Mike L. — August 1, 2006 @ 9:09 pm
Uh, maybe because Outlook Edition uses the API and syncs on a schedule automatically, even when the using isn’t doing anything?
Comment by Mike Schinkel — August 16, 2006 @ 7:44 pm
[…] As the Salesforce Heretic says, Salesforce may see a big drop in API traffic when they implement these changes. They should remember that a drop in API traffic is probably a really good thing… […]
Pingback by gokubi.com » Blog Archive » Relationships — January 7, 2007 @ 11:16 am
Apex Code for Beginners at the Salesforce.com Spring ‘07 release
The final session that I attended at the launch was Apex Code for Beginners, delivered by Adam Gross. It was a great talk, providing some nice slices of Apex Code to whet your appetite.
Adam sees the Apex Platform as based on five pillars:
Data …
Trackback by As A Service — March 26, 2007 @ 5:14 am