Conducting Data Warehouse & Business Intelligence Software Evaluations

Tuesday May 25th 2010 by Denise Rogers

Denise Rogers discusses the essential tasks in conducting effective software evaluations revolving around data warehousing and business intellegence. Each step has a dependency on the previous one, starting with establishing the framework of the evaluation and adding progressively elaborate data that facilitates a decision making process that is resolute.

Denise Rogers discusses the essential tasks in conducting effective software evaluations revolving around data warehousing and business intellegence. Each step has a dependency on the previous one, starting with establishing the framework of the evaluation and adding progressively elaborate data that facilitates a decision making process that is resolute.

So the buzz is that there is a huge and I mean HUGE initiative about a data warehousing and business intelligence program and related projects that will put the company in a great competitive position in the market place.

After the kickoff meetings, the balloons, confetti and catered breakfasts have been popped, tossed and consumed, reality sets in! The reality is that your unit has primarily provided support for transactional, retail type operations that "keep the lights on" for a number of business areas and now you are being asked to expand your support infrastructure to provide services for data warehousing and business intelligence! I don't know about my data architect family out there, but that's enough of a smack of reality to make me put down the toasted sprouted wheat bagel with smoked salmon and freshly squeezed orange juice at the kickoff breakfast!

The reason for this wave of nausea that has overcome us is that these types of projects mean building a complete infrastructure, including a service management component in support of something where nothing exists!

So we do what all data architects do. We work with the project management team and IT Support to conduct a number of extremely important software evaluations. These are career-defining moments! It's like being charged to choose the new car for the family that is not only moving from an apartment in the city to the suburbs but is also expanding to include the addition of new members to the family including a new puppy.

If you have conducted the evaluations well, you have successfully created a solution set that enables the program to take giant leaps forward to successful delivery and added some sheen and luster to your reputation as super hero! If the evaluations were poorly executed, you have placed the project at extreme risk and added to your reputation as a super zero!

So what are the essential tasks in conducting effective and impactful software evaluations? In many ways, it's very similar to purchasing a new car. Think about it! When you make the decision to buy a new car for whatever reason, it usually includes figuring out what type of vehicle, reading tons of reviews and blogs, visiting lots of car dealerships, taking endless test drives and speaking with various friends and family members in order to have enough information to make the right decision about what vehicle fits you and your family best.

Of all the evaluations I have done so far, it's not just the software itself that is being evaluated but it's the vendor and the company that is being evaluated too! There are a number of mandatory tasks and/or steps that guarantee success every single time. It's all about doing the groundwork to ensure that due diligence was done. Each step has a dependency on the previous one, starting with establishing the framework of the evaluation and adding progressively elaborate data that facilitates a decision making process that is resolute.

Understand the problem

What is the scope of the project? Will this project have ripple effects across the enterprise? There must be a complete understanding of the scope and the business requirements. What is the problem that this project is being asked to solve? The answers to these questions create the framework for the evaluation process.

Research the industry leaders and trailblazers

What software vendors are leaders in this space? What are the research groups saying? What is the market place's view of these vendors? Are these companies financially solvent? Do they have a robust organization? What is the size of their customer base? Are they perceived as a forward thinking/leading edge company? In this economic climate, these questions could not be more pertinent. This information provides the evaluation team a perspective on the fiscal strength and reputation of the leaders and trailblazers, as well as the company's image as a technological leader in their domain.

Select a list of vendors

Based on the research done in the previous step, the list of vendors is an easy one to generate. However, the list should include vendors that are leaders, up and coming companies, niche players, even open source companies.

Create and Publish the Request for Proposal

The RFP should include information related to the project scope, functions and features that the software solution must have, approaches and methodologies used as well as support and operating costs.

The vendor's responses to the RFP provides the team with enough information to refine the vendor list and initiate the process of scoring the vendor responses based on weighting factors establish by the team.

Score the selected vendors

After the vendor scores have been tabulated, ranked and rated, a picture is beginning to form that gives the evaluation team a good idea of potential software solutions that would be a good fit for the project and strategically for the company. (Whew! That was intense!)

Conduct the technical demos (aka dog and pony)

These meetings, apart from the vendor demonstrations of how many customer logos and screen shots can be included in a presentation deck and the marketing team's ability to say as many buzz words as humanly possible, are really the first chance that the evaluation team gets to see the software. When I say "SEE THE SOFTWARE", I mean paying close attention to navigation throughout the toolset, the number of steps that are required to complete common functions, wizard-like assistance, and "out-of-the-box" packages to enable solutions to be built quicker.

Conduct a number of customer reference feedback sessions

Schedule conference calls with customers that have used the software solution(s) that your company is considering. All members of the evaluation team should participate and gain a deeper understanding of how the software works within a real life situation. Also, ask the customer reference if they had a chance to do it over, what would they do differently.

Research and Review Social Media Outlets and Tech Blogs

Research the blogs and all the social media outlets. In my experience, the opinions expressed here are real and at times brutally honest. This task enriches the previous one in that real customer experiences are captured.

Review session to reduce and refine vendor list

Okay, so you are almost there and you and your time have survived! After going through the first eight steps, it should be pretty easy to know which of the vendors are serious contenders and which ones are truly not a good fit for the project and the company direction. The list should be no more than three of the top vendors.

Conduct the Proof of Concept and review the results

This is the most critical task in the effort! A successful proof of concept must start with a carefully structured test where you have the evaluation team and the vendor together on site to achieve a specific result. At the end of this task, the evaluation team should have answers to the questions, "Did the software achieve the results expected? How difficult was the software to work with? How did the software perform on the server? If there were errors, how intuitive was it to resolve the issues? Will there be a significant learning curve for the project team?"

Compile findings and create summary

Document everything! The evaluation team spent a tremendous amount of time and effort. A lot of data was recorded. This is where the team catalogs and packages information from all previous steps in the process and creates a complete summary to be presented to IT management on the software solutions evaluated. It should be crystal clear which vendor came out on top!

Make final selection and recommendation

The work is done and management decision is a pretty easy one to make because the evaluation was comprehensive and complete and would withstand any audit.

Now here are some stories of doing these steps and not doing these steps!

Should I go with the semi-tractor trailer or the golf cart? The red sports coupe is really cool!

Okay, so I am working on assignment at a client site, on a new project to completely overhaul and modernize their entire IT infrastructure. This includes silo-ed applications that had data replicated everywhere in all forms, shapes and sizes. Part of the future state environment was to consolidate and integrate all common data in one book of record for CUSTOMER! Sounds like a pretty good project to work on as a data architect (me). Well, for whatever reason between the project manager and the project sponsor the selection process took the form of listening to a vendor presentation that has sold products to this company prior to this project. That was enough for them to purchase an ETL solution (include data profiling and data cleansings components). What do you think happened? Well for starters, the installation and configuration took several weeks due to pre-requisites that were never discussed, several components would generate errors and flag the databases as corrupt, not to mention countless hours with the vendor support to troubleshoot slow responsiveness of the servers and compatibility issues. Whew! I just broke out in a cold sweat thinking of the countless traces I had to send to the vendor support group to create patches for! Needless to say, the project did not meet its project delivery dates and there were many cost overruns. And what do you think the usage of that ETL software includes now? Well the data profiling and cleansing components are either shut down or used sparingly and the ETL toolset is used at about 50% of its functionality! The project spent over $1,000,000 on a software solution that returned a fraction on its investment! Ouch! And to think if a proper evaluation had been done, the amount of pain, suffering and cost that could have been avoided! YIKES!

I want mud flaps on the pickup truck but I don't want the bed-liner!

While working for a healthcare company in the IT division, I had an opportunity to work on a DW/BI project that included ETL and BI components. This company had not had done any real data warehousing or business intelligence at the corporate level so this was a new phenomenon for everyone. We worked as a team conducting evaluations as defined in this article all the way to going through a complicated amount of red tape to obtain servers on loan to conduct the proofs of concept! In the POC however, we took it a step further and also engaged the vendors to do the installation as part of the structured tests! We really, really, really got a strong sense on how each vendor's solution worked and as a result selected the products that were a perfect fit for the project and the company! The project was completed on time and within budget. This was done a few years ago and these products have survived and are an integral part of the infrastructure today!

As you can clearly see, it really pays (literally and figuratively) to take the time to do software evaluations properly!

» See All Articles by Columnist Denise Rogers

Mobile Site | Full Site