1. Why Evaluation Matters Now
Every AI automation platform looks good in a demo. The demo runs on clean sample data, the AI answers confidently, and nothing goes wrong. The real test comes later — against your data, your systems, your volumes, and your auditors. That is where most AI projects run into trouble.
of AI projects fail, by some estimates — twice the rate of IT projects without AI.
RAND Corporation, August 2024
of agentic AI projects will be canceled by the end of 2027, Gartner predicts.
Gartner, June 2025
of enterprise generative AI pilots achieve rapid revenue acceleration; most stall with little or no measurable P&L impact.
MIT NANDA, 2025
These figures measure different things — an estimate of project failure, a forecast of cancellations, and pilots without measurable financial impact — so they shouldn't be added together or compared directly. But they point the same way.
The research agrees on why. RAND's interviews with experienced practitioners point to missing data, inadequate infrastructure, and a focus on the latest technology over real problems. Gartner points to escalating costs, unclear business value, and inadequate risk controls — and estimates that only about 130 of the thousands of agentic AI vendors are real. In other words, most failures come down to data, integration, governance, and operations: exactly the things a demo doesn't show.
The question isn't whether a platform can impress you in a demo. It's whether it will still be working — safely — a year after you've implemented it.
2. How to Use This Guide
The ten questions apply to any AI automation platform, including ours. Each question includes:
- Why it matters — the documented cause of failure it guards against
- Questions to ask any vendor — to use in meetings, demos, and RFPs
- Red flags — warning signs in demos and proposals
- How to test it in a proof of concept — on your data, with your team
- How appRules answers it — a short, clearly labeled note, so you can compare
Use the scorecard to rate each platform on the same ten dimensions, and the 30-day plan to structure a hands-on evaluation.
3. The Ten Questions
- 1. Real data access — read and write
- 2. Governance of AI actions
- 3. Integrity at design time
- 4. Verifiable AI answers
- 5. Tamper-proof audit trail
- 6. Production scale and control
- 7. Security and access
- 8. Interoperability and openness
- 9. Who can build and maintain it
- 10. Substance over agent washing
Can it reach your real data — reading and writing, not just reading?
Why It Matters
RAND's research found that AI projects often fail because organizations lack the data an effective model needs, or the infrastructure to manage it. A platform that can only read a few cloud apps through generic API calls will stall the moment a workflow needs to update an ERP record, reconcile a legacy database, or move millions of rows.
Questions to Ask Any Vendor
- Which of our systems can you connect to today — and can you write to them, not just read from them?
- Do your connectors understand each system's structure: entities, relationships, required fields, and valid values?
- How do you handle large volumes, legacy databases, and on-premises systems?
- What happens when a system isn't supported?
Red Flags
- Connectors that are a single generic node per API, with everything else left to custom code
- Demos that only read data and never write it back
- “We support 1,000+ integrations” with no way to check depth
How to Test It in a Proof of Concept
Pick two systems that matter most — one cloud, one legacy or on-premises — and have the vendor read, transform, and write back real records, including a field with a restricted list of valid values.
appRules connects through metadata-driven, full read/write connectors across 36 categories — with live entities, relationships, and picklists — and every connector brings its own activity library. Enterprise data integration · Integration Connector Directory
Does it govern what the AI is allowed to do?
Why It Matters
Gartner cites inadequate risk controls as a leading reason agentic AI projects will be canceled. When an AI model can call tools or write data directly, every model error becomes a business error.
Questions to Ask Any Vendor
- Can the AI change data directly, or does every action go through validated, predefined operations?
- How are AI outputs checked before they're acted on?
- Can you restrict which systems, fields, and actions an AI-driven workflow may touch?
- What happens when the AI returns something unexpected?
Red Flags
- “The model is very accurate” offered as the governance strategy
- Guardrails that only filter text going into and out of the model
- No way to see, before a run, what the AI is permitted to do
How to Test It in a Proof of Concept
Feed the workflow a deliberately bad input — an out-of-range value or an invalid category — and watch what the platform does before anything is written.
In appRules, AI advises and the platform acts: AI output passes through deterministic Conditions, validated Actions, and post-conditions before anything changes. GuardRailed Automation · Contain the Model
Does it catch errors at design time — before they're built?
Why It Matters
The cheapest error is the one that's never built. Platforms that generate configurations from free-form prompts can produce workflows that look right but reference fields that don't exist or values that aren't valid — leaving someone to review every line.
Questions to Ask Any Vendor
- Are fields, values, and mappings chosen from the live schema, or typed by hand?
- Is each property validated as it's configured?
- Is the whole workflow validated before it runs — and will a scheduler run an invalid one?
- If a configuration is generated from a prompt, how is it verified?
Red Flags
- AI-generated workflows with no validation step
- Validation that only happens when a run fails
- Free-text fields wherever a controlled value belongs
How to Test It in a Proof of Concept
Build a small workflow yourself during the trial. Try to map a field to the wrong type, leave a required property empty, and schedule the incomplete workflow — then see what the platform catches, and when.
appRules channels design through guided, metadata-driven dialogs, uses the Context Picker in the SmartPrompt Builder, validates every property as it’s configured and every workflow before it runs — and the scheduler won’t run an invalid workflow. GuardRailed Design for AI Workflows
Can you verify AI answers rather than trust a single model?
Why It Matters
No single model should be the sole basis for an important decision. Models differ, drift, and occasionally fail outright — and when one model is asked to check another, the problem has only moved.
Questions to Ask Any Vendor
- Can a decision be checked against more than one model?
- What decides the outcome when models disagree — another model, or a rule you define?
- Can you inspect exactly what the AI was sent before it is called?
- Can you choose — and change — the model for each step?
Red Flags
- Lock-in to a single model provider
- “Our AI checks its own work” as the verification story
- No way to see the prompt and context the model actually received
How to Test It in a Proof of Concept
Run the same decision through two different models on a set of records you've already labeled, and compare. Then check how the platform handles the records where they disagree.
appRules Model Cross-Validation runs multiple models, and deterministic Result Selection and Deadlock Resolution rules — including a fail-safe — decide what their answers mean. Partial Preview shows the merged prompt before any AI call. Model Cross-Validation
Is there a complete, tamper-proof record of every run?
Why It Matters
Regulated industries must show why a decision was made. Without a complete record, a single disputed outcome can stall an entire program.
Questions to Ask Any Vendor
- What exactly is logged for each run — inputs, AI calls, decisions, and actions?
- Are logs tamper-proof?
- Does the AI record its reasoning in a form a reviewer can read?
- Can you export the evidence for an auditor?
Red Flags
- Logs limited to errors
- Records that can be edited after the fact
- No link between an AI call and the action it led to
How to Test It in a Proof of Concept
After a test run, ask the vendor to reconstruct one specific decision end to end: what data was read, what the AI was asked, what it answered, and what was written.
appRules keeps tamper-proof logs and statistics for every run — activities, data source operations, AI model calls, tokens, condition results, and actions — with AI reasoning notes and CSV export. Logging & Statistics
Will it run reliably at production scale — and can you stop it?
Why It Matters
MIT found that most generative AI pilots stall without measurable financial impact. A pilot that works on a hundred records proves little about a nightly job over millions — or about what happens when something goes wrong mid-run.
Questions to Ask Any Vendor
- How do you schedule, distribute, and monitor long-running work?
- What happens if a job fails halfway through?
- Can an authorized person pause or stop a run mid-task?
- How do you scale — and what infrastructure do we have to manage ourselves?
Red Flags
- Scaling that requires you to assemble and operate the infrastructure
- No way to stop a run that is already in progress
- Pricing that grows with every execution, making high-volume work unpredictable
How to Test It in a Proof of Concept
Run a realistic volume — not a sample — and, partway through, ask the vendor to pause and then cancel the job. Note what you can see and control while it runs.
appRules runs on multiple job servers with shared job groups, load balancing, and caching, supports runs lasting days or months, and lets authorized users pause or cancel any running job from the Admin Center. Large Scale & Long-Running Jobs
Is it secure — and does it fit how you manage access?
Why It Matters
AI automation touches sensitive data and credentials. Security that depends on everyone being careful is not security.
Questions to Ask Any Vendor
- Can sensitive connections, settings, and workflow components be locked to authorized people?
- Does it support sign-in with your organization's identity provider?
- Where can it run — in your cloud, on your premises, or both?
- How are credentials stored and used?
Red Flags
- Credentials embedded in workflow definitions
- All-or-nothing access to the design environment
- A deployment model that conflicts with your data residency rules
How to Test It in a Proof of Concept
Ask a colleague without permissions to open a protected connection or workflow. Confirm what they can — and can't — see.
appRules locks sensitive design-time records and metadata with the Secure button, supports sign-in with organizational credentials through OpenID Connect, and runs on Windows desktop, Windows Server with IIS, or Azure. One Secure Button · Comprehensive Microsoft Integration
Does it work with the tools you already use?
Why It Matters
No platform lives alone. Lock-in to one model, one cloud, or one assistant limits your options as the market changes.
Questions to Ask Any Vendor
- Which AI providers and models can we use — including local or private models?
- Can AI assistants and agents call governed workflows through open standards such as MCP?
- Can workflows call external REST APIs?
- How does it coexist with automation tools we already own?
Red Flags
- Proprietary integration paths only
- A single model provider
- “Rip and replace” as the only adoption path
How to Test It in a Proof of Concept
Connect one tool your team already uses — an AI assistant, an agent framework, or an existing automation platform — and have it trigger a governed workflow.
appRules supports 20+ AI providers including local models, exposes governed workflows through the appRules MCP Server, calls external REST APIs, and is designed to work alongside tools such as Power Automate, LangChain, and Visual Studio. AI Provider Ecosystem · Product Interoperability
Who can build with it — and what does it really take?
Why It Matters
If only scarce specialists can build and maintain workflows, every change waits in a queue. If only beginners can, complex requirements hit a wall.
Questions to Ask Any Vendor
- Can business and IT teams build without code — and can developers extend with real code when needed?
- Are no-code and code mixed within the same workflow, or separate products?
- How are workflows versioned, tested, and moved between environments?
- What does it take to maintain a workflow a year from now?
Red Flags
- “No code” that ends at the first real requirement
- Code that lives outside the platform's governance
- Every change requiring the vendor's professional services
How to Test It in a Proof of Concept
Have one business analyst and one developer each build part of the same workflow during the trial, then swap and modify each other's work.
appRules combines no code, low code, and source code on one platform: business and IT teams build in the designer, and developers extend in Visual Studio with any .NET language — and custom building blocks become available to no-code builders. No Code, Low Code, and Source Code · appRules + Visual Studio
Is the capability real — or is it agent washing?
Why It Matters
Gartner estimates that only about 130 of the thousands of agentic AI vendors are real, and warns of “agent washing” — relabeling existing products as agentic without real capability. A label is not evidence.
Questions to Ask Any Vendor
- Can we see the actual product — not slides or mockups — doing what you describe?
- Can we test it on our own data, in our own environment?
- Which customers use it in production, and for what?
- What exactly is generally available today versus on the roadmap?
Red Flags
- Demos that only run on the vendor's sample data
- Capabilities described only in marketing language
- Reluctance to allow a hands-on trial
How to Test It in a Proof of Concept
Insist on a proof of concept on your own data with your own team at the keyboard. A vendor confident in its product will welcome it.
appRules publishes real product screenshots, its full workflow activity library, complete connector directories, and fair product comparisons — so you can check every claim before you talk to us. Why the appRules.ai website is so big
4. The Evaluation Scorecard
Rate each platform from 1 (weak) to 5 (strong) on every dimension, using evidence from your proof of concept rather than the vendor's presentation. Any dimension scoring 1 or 2 deserves a hard look before you commit — a single weak link, such as governance or data access, can stall an entire program.
| Dimension | What “Good” Looks Like | Score | Evidence / Notes |
|---|---|---|---|
| 1. Real data access — read and write | Full read/write on your real systems, with schema and valid values understood | 1 2 3 4 5 | |
| 2. Governance of AI actions | Every AI-driven action validated and constrained before it executes | 1 2 3 4 5 | |
| 3. Integrity at design time | Choices made from what is valid; every property and workflow validated before running | 1 2 3 4 5 | |
| 4. Verifiable AI answers | Multiple models, with rules you define deciding the outcome | 1 2 3 4 5 | |
| 5. Tamper-proof audit trail | Every input, AI call, decision, and action recorded and exportable | 1 2 3 4 5 | |
| 6. Production scale and control | Proven at your volumes, with live monitoring and the ability to stop any run | 1 2 3 4 5 | |
| 7. Security and access | Sensitive components locked; organizational sign-in; deployment that fits your rules | 1 2 3 4 5 | |
| 8. Interoperability and openness | Model choice, open standards, APIs, and coexistence with your current tools | 1 2 3 4 5 | |
| 9. Who can build and maintain it | No code through real code on one platform, with governance throughout | 1 2 3 4 5 | |
| 10. Substance over agent washing | Real product, your data, your team at the keyboard | 1 2 3 4 5 |
5. A 30-Day Proof-of-Concept Plan
A short, structured trial on real data tells you more than months of presentations. A simple plan:
Week 1 — Connect
- Pick one real business process and two systems — one cloud, one legacy or on-premises
- Connect both; confirm read and write
- Agree on success criteria and the volume to test
Week 2 — Build
- Have your own team build the workflow, with the vendor coaching
- Include one AI decision and one write-back
- Note what is caught at design time
Week 3 — Break
- Feed bad inputs and edge cases
- Compare AI answers across models on labeled records
- Run at realistic volume; pause and cancel mid-run
Week 4 — Prove
- Reconstruct one decision end to end from the logs
- Review security and access with your security team
- Complete the scorecard with evidence
6. Conclusion
AI automation can transform how an enterprise operates — but only if the platform underneath it can reach real data, govern what AI does, catch errors before they reach your systems, prove what happened, and keep running at scale. Those qualities don't show up in a demo. They show up when you ask hard questions and test on your own terms.
We wrote this guide to be useful whichever platform you choose. We also believe appRules answers every one of these questions well — and we'd welcome the chance to prove it on your data.
Don't buy the demo. Test the platform.
- RAND Corporation, The Root Causes of Failure for Artificial Intelligence Projects and How They Can Succeed (RR-A2680-1), August 2024.
- Gartner, "Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027", June 25, 2025.
- MIT NANDA, The GenAI Divide: State of AI in Business 2025, as reported in Fortune, August 18, 2025.