Bulk API: Pumping Up Data Movement

Forget everything you know about bulk APIs. Cloud Elements is changing the game.

Feel like you need more muscle when it comes to moving data into or out of your application in bulk? You’re not alone. Most endpoint service APIs available today don’t offer native resources designed specifically for bulk data movement, meaning you’re left to bulk up on your own... or have Hans and Franz help you.

Image Credit: cinemablend.com

Why Do I Need Bulk APIs?

At Cloud Elements, we’ve seen firsthand an ever-increasing demand for the ability to move data in bulk. Luckily, we’re very experienced in this area — entire LOBs rely on our Bulk capabilities so they can offer bulk data movement to their customers.

Here are some typical use cases where bulk API plays a critical role:

1. SaaS Applications: A prime example where bulk data movement is essential occurs when a new customer signs up. After signing up, the app needs to do an initial bulk load of data to complete the initial processing of that data set. In other words, it needs to move all contacts in the customer’s CRM so they can be processed and extended. Once processing is completed, those contacts, along with their now extended data fields, are moved back into the source system. From there, the contacts can be updated in real-time using event notifications and direct API calls.
2. Daily Data Sync: Another circumstance where the movement of large quantities of data is necessary is during your daily data sync. For example, once a day you might sync all invoices and payments in one service with purchase orders in another service. This requires that you have adept bulk data movement capabilities so that information is conveyed quickly and accurately.
3. Backups to Databases: Additionally, you might need bulk API to assist with backing up daily transactions from a finance system into a secure data warehouse, such as Snowflake. Bulk data movement can also help you transfer records from your financial program to an on-prem database such as mySQL or SQL Server.

Why Is Bulking Up So Difficult?

Sadly, most endpoints don’t offer bulk data APIs to make the process easy. This means developers have to get creative and work around the complexities that come with building and maintaining a bulk data API framework.

Of the hundreds of Elements in our public catalog, personally, I can only think of four or five that do offer native bulk APIs: Salesforce, Eloqua, Netsuite, and Marketo are examples. However, even these bulk APIs have limits. For example, Salesforce limits batches of data to 10,000 records (along with other limits), so as a developer looking to move data in bulk, you need to chunk the data into proper size limits and multi-thread the requests. This adds just another step to your developers’ long list of tasks to complete.

As a Product Manager, it’s your duty to estimate the costs and gauge the time it takes to build out bulk services in your product so you can accommodate your customers’ bulk use cases. This is harder than you think, as you’ll need to account for each endpoint’s search query differences, data encryption and security, endpoint data limitations, data paging, and error handling, just to mention a few. It’s going to be expensive — brace for it.

How Is Cloud Elements Changing the Game For Bulk API?

You’ve probably seen it in the news before — a professional actor lands a coveted role, spends hours with a trainer, vastly changes their diet, and constantly tracks their results to achieve a desired physique. Well, that’s basically what we did at Cloud Elements - we bulked everything up, made necessary changes, and tested their effectiveness to make our product, Bulk V3. 

Cloud Elements is the only solution on the market that provides a full-featured bulk framework by default as a part of each public Element. So even for APIs that don’t offer bulk natively, our Element enhances what you as a developer can easily do with that endpoint. Not only that but if you want to customize an Element to meet your specific needs, you can easily do so with our Element Builder. Additionally, we have VDRs that can handle the mapping and transformation of data as it arrives from your bulk requests.

With our enhanced Bulk V3 service, we’ve taken all of the best components from V2 and made them better, stronger, faster, and bigger.

So what exactly is inside Bulk V3? Here are just a few of the bulked up features:

1. New UI dashboard and wizard for no-code usage of Bulk V3 features
2. New, separate microservice to increase speed and scalability
3. Immediate callback when bulk request is complete
4. Direct write of results to cloud storage services such as Box, Dropbox, and Google Drive
5. More control of result sets with new “limiter” settings. This comes in handy for working around endpoint services that have daily API call limits
6. Simple query language, based on SQL, to filter data requests
7. Works automatically for every Elements “GET” resource (e.g. GET /contacts) where the endpoint supports paging — even resources that you’ve extended in the Element!

An added bonus of Bulk V3: we even made it easier to try out and use via a new UI!

Already using our amazing Bulk APIs? No worries, V3 is backward compatible with the previous version. We do, however, encourage you to convert to the newer Bulk V3 resources soon — mostly because they’re easier to use, highly capable, and, frankly, because they’re the future of bulk data movement.

Fun fact: Bulk V3 is so huge, we code-named the project “Hulk”, but left out the anger issues.

To Wrap It Up

Bulk data movement can be a huge headache, which is why we created Bulk V3. Want to learn more about how this service could benefit your business? Drop us a line, hire us as your trainer, and Bulk Up today - it’ll be a smash for your business.