Your coming from a late day meeting when you return to the flashing red light on your phone blinking like crazy. As you look at your phone it says 32 missed calls. Thats odd, I normally dont get that many calls in a day. As you pick up the phone to hear the message while logging back into you work device, you see Slack posts directed to you as well as your ALL CRM group. One of the tech lead from Dev Ops has slacked you that they are getting errors on their integration API account in the syslogs. That odd you then get a phone message, the VP of Sales leaves a message “My reps cant get incoming leads or dial out, were losing money here and its almost end of month!” What is going on down there? The second message is one of two Customer Service Directors stating we cant log in our calls into the CRM, Salesforce is down! I repeat Salesforce is down!
OMG you think, let me jump on truth.salesforce.com I have a major outage issue here. How in the world did this happen? What happened to the multi-tenant architecture thats supposed to stop this from happening? You sit there a little confused for a few minutes. Salesforce is reporting no system outages on your instance. After further investigation and the IT team escalating the issue to your CIO, you realize that the impacted users are all using externally integrated systems. Could it be that something happened to the integration accounts?
Looking deeper it sure was, your externally connected apps have consumed all the allocated API Calls for the Salesforce Instance.
Here is how you manually check your API Usage –
- Click Setup
- Company Profile
- Company Information
- ‘API Requests, Last 24 Hours’
You will see on the Company Information tab “API Requests, Last 24 Hours” Here you will see your number 300,000 (300,000 max). The screen shot is my personal dev org which shows 0 out of the 15,000 provided.
Your company API requests are specific to your Org so the Max number can be higher or lower.
You noticed, yes one of your external API connection has in deed consumed your API calls for the rolling 24 hours. This part is very important, API usage is rolling in 24 hours. So, if for some reason a big job was run at 9 pm the day before, once it becomes 9 pm today those used api calls roll back into your allowance.
Once you identify that the API limits were maxed, you should do a couple of things. You should report it up the chain to your management and executive suite of the issue. You should also open a case with Salesforce to see if they will allow you a temporary API increase to get your org back up and running ASAP. This normally needs high level approval on the Salesforce side and your Sales Rep will be contacted of the situation. They will only do this once so you will need a long term plan to finding out whats consuming all your API allocation.
You next step is to find the culprit. Just an FYI, normally the 1st person who called to complained is your primary suspect but we need more than a sneaky suspicion. The good news is Salesforce stores that data on your Org for you to check. You just need to learn how.
- Click on the Reports Tab
- Look for the Salesforce provided ‘Administrative Reports’ folder. The one you cant delete reports from or add to.
- The first report on the list will be called ‘API Calls Made Within Last 7 Days’
After running the report, you will want to customize it and group the report by Username. Remember it gives you the last 7 days so your looking for the day that you actually consumed all your API calls. You now found your offending user account and can report back to that team to see if they can make the calls more efficient when they query your Salesforce Org.
*Pro Tip: while you now know that this report exists, Schedule it to run daily and send you the last 7 days of API calls so you can monitor it for the next week or two for trends. As the owner of a Salesforce Org this should be on your radar at all times.
One of the major issues with the above is that Salesforce will only record the last 7 days in our instance. This can be an issue if you wanted to see utilization from an API account a few weeks, months or years ago. Its also difficult to see trending, e.g. if one of the external systems get a large import of data quarterly that syncs to Salesforce. You will miss these very important stories in your data.
Long Term Fix
So, if you want to track this information for more than 7 days, you are going to need to pull it to an external source using the Salesforce API. This can be done so many different ways. One of the companies I worked at used Omniscope and pulled the data to a SQL database. Another company used the Streaming API with a tool called SteamSets and dumped the information into Grafana. You can also use Jenkins jobs to query the database and dump it into another Salesforce Org if needed. So at this point you should look at what tools you have in your organization and what developers you have as a resources.