For most companies, IT-related hardware and software maintenance is costly, time-consuming and requires hiring and retaining a support staff of operating system and database management system specialists. Delegating these responsibilities to an outside firm allows a new application to be developed and implemented more quickly. Enterprises also consider this possibility as a way to purchase only the resources that they need at initial implementation. As the application data and number of users grows, additional IT resources can be purchased as needed.
However, there are issues with delegating database support to an outside service. In this article, we will concentrate on how these environments scale with application growth, especially if the data and services are stored in the cloud.
The Cloud and DBaaS
The cloud is a general term meant to express the idea that certain aspects or functions of an application or its data are executed or stored non-locally. Commonly a company would create a local data center, with accompanying lighting, staff, hardware and software. A service provider would create and maintain one or more of these sites at their own premises, and connect everything to the company via a private network or even the Internet. As a result, the company is relieved of the responsibilities and costs of maintaining their own site and equipment. It also provides a way to rent or lease only the resources the company needs for initial rollout of its application. As the users and data grows, they can purchase more services.
Database as a service takes this a step further. In addition to providing the site(s) and hardware noted above, the DBaaS provider also provides database management software, data storage, and database administrators.
The Emphasis on Fast Application Development
Enterprises want to minimize development times for new applications. One way to do this is to delegate or outsource various development or production processes. One of these is the creation, tuning, and maintenance of the application database environment, known as Database as a Service (DBaaS). The theory is to have an outside organization create the hardware environment, install the operating system and database management software, and handle upgrades and performance tuning. DBaaS implementations are commonly done in the cloud.
How Applications and Databases Scale Up and Out
The phrase scale up is used to describe several different areas of application and database growth. Each of these requires different expertise to detect and address, especially if the support staff are remote and not directly connected to the enterprise that developed the application. The areas of database administration affected by growth are usually the following:
- Database backup and recovery;
- Purge of old or unused data;
- Analyzing data for trends;
- Capacity planning and performance tuning.
Let’s review various growth categories and how they affect database administration. Keep in mind that with DBaaS the enterprise has delegated or outsourced database administration to another organization, and that the data is typically stored in the cloud.
More Users and Higher Transaction Volumes
Many applications are written in order to sell products or services. These applications depend upon customer transactions in order to gain back in profits the costs that were expended in application development and implementation. If more users buy more things, profits should rise; therefore, there is an expectation that the number of users and numbers of customer transactions per day will increase over time.
Normally the database administrator (DBA) will meet with the developers to plan ahead for this kind of growth. In the case of DBaaS , these steps may be skipped or glossed over, with the DBaaS provider assuming that they can accommodate this growth by adding more hardware CPU power, more memory, or more disk space. Regrettably, as noted above, several other DBA tasks get forgotten. As users and transactions increase, the time available for database backups decreases. In addition, performance tuning of the database management system becomes more important and costly.
Growing Amount of Data Stored
Obviously, as the number of customer increases and the number of products and services offered increases, the amount of data stored in the database grows. More disk space storage is required, and database backups take longer. In case disaster strikes and the DBaaS provider needs to recover the database, the larger the database the longer the recovery period to re-load all the data.
As the application gets more use and the customer base grows, the business discovers that it can analyze the data to discover sales trends, develop customer profiles and get ideas for better customer service. Are certain products popular with customers in a certain geographic area? Are some services purchased by new customers but not repeat customers? Are certain combinations of products purchased together? Are some services more popular during some seasons but not others?
All of these correlations and more can be derived by applying business analytics to the data, and the more data you have the more accurate the forecasts. The application owner wants more data stored for analysis, while the DBaaS provider will want less data stored so that database access is faster and database backups are smaller and run faster.
One final issue with data growth centers on capacity planning. It is fairly easy to estimate how much additional media storage is required to retain data. Twice the number of customers means that the customer database will be twice as big. However, this assumes that the application owner’s capacity requirements don’t change. For example, an owner may specify that they will expect 10,000 initial customers in the database, that growth will be 25% per month, and customer data will be retained for one year.
What is commonly forgotten is that data retention requirements commonly change for unexpected reasons. In our example, let us assume that the application owner allows customers to pay by credit card. Unfortunately, after a few months of operation they are informed by their credit card processing provider that they are legally required to retain seven years of customer transaction data. This more than triples the projected storage requirement (two years to seven years) and while this additional storage will not be required immediately, the additional costs will reduce profits.
Adding Application Features and Functions
Applications are rarely static. They change and grow for a variety of reasons. In today’s application development environment, it is popular (and faster) to rollout only basic functions at first. There are several reasons for this:
- Users will perceive a simple application as being easier to understand and use;
- A small number of users will exploit advanced or complex features, so these are relatively costly to implement;
- A basic application is quicker to develop, quicker to implement, and will most likely have fewer errors or issues;
- Should the application prove to be unpopular or too costly to maintain, developers will have spent less time and fewer resources.
After initial implementation of a successful application it is a natural progression to add more features and functionality. Along with more application code and more stored data there is a corresponding expansion in the data model.
Changes in Data Structure and Complexity
As the application grows, new data elements are added. The product database may expand to include product images, the service database may require audio or video clips of satisfied customers, and data transferred to other applications may require a special format such as extended markup language (XML). Indeed, sometimes a major application enhancement requires a redesign of most of the database.
This kind of growth is extremely hard to predict. How will the application owner and DBaaS provider develop a cost model for these changes? Who will be in charge of the enterprise data model changes, who will make decisions about new or changed business rules, and how much will the application owner be charged for any of these services? In addition, a new data model or new data types may affect database backup processes or backup times, especially if the application now requires relatively large data items such as video files or scanned documents. How long must this data be retained? How and when will it be purged? These and other questions need to be addressed early if a DBaaS relationship is to succeed.
New applications take time to develop, and an enterprise may fear that an extended development time will cause rollout to be delayed, thus losing potential customers or allowing competitors to implement first. DBaaS removes substantial costs and resources from application developers, allowing them to develop and implement more quickly. However, issues connected with scaling up need to be addressed early by the application owner and DBaaS provider.
Growth in the number of users and transaction volumes requires DBAs to monitor backup and recovery requirements as well as upcoming increases in computer power and storage. As the application matures, the amount of customer historical data grows to the point where a new business analytics application may be deemed valuable. Application upgrades and new features will create changes in the enterprise data model, leading to new business rules and decisions on how to implement them. New data types may be required that need to be taken into account during backup and recovery.
These changes are difficult to predict during the initial discussions and negotiations with your service provider. Make sure that you raise these issues early on, so that you won’t be surprised by costs that may well spell an end to a profitable new application.