Blog | Hatch

12 Questions to Ask When Vetting an AI Tool

Written by Kristen McCormick | June 15, 2024

When choosing an AI tool, it's crucial to ask the right questions to ensure it will effectively addresses your needs, works smoothly with your existing systems, and delivers measurable ROI.

In this post, we'll explore 12 essential questions to ask when vetting an AI tool, covering aspects such as data requirements, integration ease, customization options, and more. 

And while we're at it, we'll answer those questions for Hatch AI. 

Table of contents

Here's the full list of questions we'll be covering:

  1. What specific problem does the AI tool solve, and how effectively does it do so?
  2. What type of AI does the tool use?
  3. What data does your AI require?
  4. How does the AI tool integrate with existing systems and workflows?
  5. How does your AI product achieve and measure ROI?
  6. Can your AI product be set up, managed, and optimized by anyone or does it require certain skills?
  7. What are the customization options?
  8. What kind of support do you offer?
  9. What is the total cost of ownership of your AI system?
  10. What are the experiences and feedback from current users of the AI tool?
  11. How does the AI learn?
  12. What are you doing to stay innovative and responsible?

12 questions to help you find the best AI tool for your business

By asking these questions, you'll be better equipped to make an informed decision and select an AI tool that aligns with your business objectives and enhances your operational efficiency.


1. What specific problem does the AI tool solve, and how effectively does it do so?

Identify the specific problem the AI tool addresses and the tangible benefits associated with it, and ask for metrics, case studies, and user testimonials that speak to these benefits.

How does Hatch answer this? Hatch AI solves for ineffective and inefficient customer communication, seen through low customer response rates and slow representative reply times. Its ultimate benefit is increased customer satisfaction and conversion, higher revenue and ROI, and lower overhead costs associated with operational inefficiencies. Our case studies and in-app reporting speak to this, and we also publish blog posts like this one: How Hatch Helps You Grow Revenue

We'll cover more data around these benefits in later sections.

2. What type of AI does the tool use?

Make sure you understand the type of AI that your provider uses—generative, prescriptive, or predictive. This will help you orient yourself to the tool and set the right expectations. 

You may also want to go a level deeper and ask which type of technology it uses, such as deep learning, reinforcement learning, etc. If you don’t understand the answer, ask them to explain it to you in simpler terms. A provider with true AI expertise will have no problem with this. You can also use our guide to AI to familiarize yourself more.

As for Hatch's answer - Hatch AI uses generative AI, which means it focuses on creation, whereas predictive and prescriptive AI focus on forecasting and decision-making. Generative AI creates new content or data, such as text, images, or music, based on learned patterns.

3. What data does your AI require?

This will help you understand whether your current tech stack and data will be adequate enough for the bot to work optimally and validate your investment.

This includes the type and quality of data, such as structured data, unstructured text, images, or real-time data streams. Ask also about the volume of data required and any specific data formats or sources.

For example, a customer sentiment analysis AI might require historical customer reviews and survey responses. Or an AI reporting tool may have a minimum data requirement in order to produce meaningful insights.

And here is Hatch's answer: Hatch AI requires contact data in order to launch campaigns and produce targeted messaging. The two main sources of this data are your lead sources, such as your website forms or purchased lead providers, as well as your CRM. The more complete your CRM data, the better targeting you can do with your campaigns, and the more accurate reporting you can have on your ROI.

4. How does the AI tool integrate with existing systems and workflows?

Smooth integration ensures the AI tool can enhance your existing processes without requiring significant changes or additional resources. Assess how seamlessly the AI tool integrates with your current systems and workflows. This includes what apps and tools it integrates with, as well as how easy it is to configure those integrations. In addition, look for API availability, integration guides, and support.

As for Hatch's answer, Hatch AI integrates with your lead sources so that you can engage them instantly, as well as with your CRM so that you can segment audiences and tailor your messaging accordingly. The integrations are available in our marketplace and are self-serve, but there are also onboarding and support team members who are happy to help.  

5. How does your AI product achieve and measure ROI?

Assess how the AI tool achieves and measures ROI by examining its impact on key performance indicators (KPIs). Ask for specific metrics, such as cost savings, increased revenue, or productivity gains. 

Once those are established, find out if and how this is reported on within the platform, and ask about case studies that demonstrate these outcomes.

Here's what that looks like with Hatch AI:

Hatch AI achieves ROI by improving conversion and operational efficiency, which ultimately results in higher revenue and lower overhead costs. As such, our reporting metrics include:

  • Campaign metrics: Response rates, opt-out rates, and channel performance.
  • Rep performance: Response times and output
  • Bot performance: Reply times and success rates
  • ROI: Appointments, sales, and revenue 

And as for case studies and reports:

  • Our data shows that Hatch bots have an average reply time of 5 seconds.
  • Case studies like this one show a savings of $166k per year in overhead costs with Hatch AI.
  • This case study shows that with Hatch AI sharing the work, one inside sales rep was able to cut down his reply time from 15 minutes to 2. 

Go to the case study.

6. Can your AI product be set up, managed, and optimized by anyone or does it require certain skills?

Understanding the skill requirements will help you ensure you have the right resources or support for effective implementation and ongoing management. Ask about the necessary technical expertise and training provided.

For example, a marketing automation AI might be designed for marketers with minimal tech skills, offering intuitive dashboards and drag-and-drop features.

Conversely, a more complex tool like an AI-powered data analytics platform may require data scientists to manage and optimize its use.

Here is Hatch's answer:

Hatch AI can be used by anyone with minimal tech skills. It doesn't come without a learning curve, but this is inevitable with any AI. It also takes some up-front work, but this is a healthy requirement for any customer-facing AI tool with advanced customization.  It's important that you take the time to train it just as you would a human. The difference is, they learn much faster and can be cloned.

Hatch AI includes full white glove onboarding, in addition to ongoing support. It also comes with built-in tools like our AI Analyzer that help ensure you're optimizing your bot at every step.

Our AI Analyzer grades your prompts and makes suggestions for improvement.

7. What are the customization options?

AI learns and refines itself with more usage, but it’s important to be able to have control and customization options. Find out what the “out of the box” solution entails and then take some time to think through scenarios where you’ll want to customize. Ask about these scenarios and to the point above, find out if that’s something that can be done on your own or if it requires assistance.

Hatch AI offers training and customization unlike any other tool on the market. Your agent has a name, personality, and business directive, and it follows your processes and scripts, takes the appropriate admin action, and stays true to your brand voice. 

Head here to see how Hatch AI and customization works.

8. What kind of support do you offer?

A good AI solution will provide support throughout your customer journey. This includes choosing the right solution for your business, onboarding and training, and then any ongoing support. Find out how support is provided—is it through phone, email, live chat? And do they have a knowledge base? Are they regularly publishing content? Regular meetings?

The Hatch answer?

Hatch offers support at every step of the way. Sales will help you identify the right solution for your business goals, Onboarding will make sure you're set up properly, and Customer Success will provide ongoing support through email and virtual meetings. You also have access to in-app chat support as well as our Knowledge Base and Free Resources.

9. What is the total cost of ownership of your AI system?

Find out all costs involved, including initial setup, ongoing usage, and if support is included. Make sure you also have a clear understanding of the pricing structure, especially if it is usage-based.

Hatch pricing is usage-based, meaning there is a platform fee and then additional fees based on conversation volume. So while your monthly payment will be unique to your business, every plan comes with:

  • Campaign automation
  • Message templates
  • Custom targeting
  • Shared workspaces
  • ROI reporting

The goal is to make it affordable and scalable for your business. You can learn more in our pricing page.

10. What are the experiences and feedback from current users of the AI tool?

Look for testimonials, case studies, and reviews from other users to gauge satisfaction and effectiveness. Ask for case studies and references applicable to your business and industry.

We have provided our case studies and testimonials in Questions #1 and #5, but if you want a personal reference, we are happy to provide!

11. How does the AI learn? 

In other words, does it pull from publicly available datasets, proprietary data, or customer-provided information? 

For example, an AI tool for fraud detection might use historical transaction data from multiple financial institutions and be updated quarterly with new fraud patterns. Understanding the data sources and update processes ensures the AI tool adapts to changing conditions, maintains high performance, and continues to meet your business needs effectively.

Hatch AI bots pull from a mix. They use publicly available content to be able to speak conversationally; they use proprietary algorithms we've created, based on AI best practices, in order to provide tips on optimization such through our AI Analyzer; and they use what customers input in their scripts, instructions, and business profiles to be able to speak accurately about the business and stay on-brand.

Your Business Profile within Hatch AI is an example of customer-provided data, which the bot references when customers ask them questions about your business.

12. What are you doing to stay innovative and responsible?

AI is constantly changing. Make sure your vendor is aware of trends and proactively considering security, ethics, and compliance in its AI product. You may also want to ask what their approach is around staying on top of larger trends and incorporating them into their product roadmap.

At Hatch, our AI engineers are always staying up to date on the most current LLMs (large language models) and AI prompting best practices.

And as a communication platform, we take compliance very seriously. Requirements and best practices are built into the onboarding process as well as our platform. We also provide several resources around this topic, such as this Complete TCPA Guide and Checklist.

Use these questions to find the right AI tool for you

Selecting the right AI tool involves thorough evaluation and asking critical questions to ensure it meets your business needs. By addressing these 12 key questions, you can better understand the tool's capabilities, integration potential, data requirements, and ROI impact.

Remember, the right AI tool can be a game-changer for your organization. Take the time to vet the best tools for you so you can feel confident in driving sustainable success.