Jev by TypeSafe AI: our implementation experience and results
Jev AI is a decision model that scores, classifies, and routes information against criteria you define. It returns structured results your software can use directly, not written answers. That makes it a fit when a workflow needs to decide which records qualify or deserve attention first.
Our recruiting AI assistant often suggested candidates who missed core requirements, leaving recruiters to re-check profiles and correct the shortlist manually.
We integrated Jev AI into our recruiting tool to evaluate candidates against client briefs. Production shortlist precision rose from 44% to 74%, reaching 92.5% in a filtered evaluation of candidates Jev rated as likely fits.
Here is what we changed, how we tested it, and how we handled candidate data.
Before Jev AI: how our RAG-based assistant shortlisted candidates
We provide outstaffing and outsourcing development services, with specialists in Ruby on Rails development services, React, React Native mobile development, and a wide range of AI integration, from RAG to AI agent governance services. To help our recruiters match the right people to each client request, we built a recruiting assistant in Slack.
A recruiter posts the client brief in a Slack thread and types find. The assistant returns a shortlist with one row per engineer, a fit explanation, and potential concerns.
The table is built from our own data. The pool holds 111 active software engineers. Each has interview reports written by our engineers after a technical interview, and often a résumé too. Together that is about 760,000 characters of text.
A recruiter reads five rows, maybe ten. So the order of the table matters as much as its content.
Search based on RAG
The first version ran on RAG built on an AWS Bedrock Knowledge Base. It followed the standard path for RAG development services: connect the data sources, configure retrieval, and check what the search returns. We used the managed type of Knowledge Base, so AWS runs the search infrastructure for us, and we don’t have to build or maintain it.
Data preparation
Every new interview report or résumé goes into Amazon S3 storage. The Knowledge Base then splits each document into short pieces of text and converts each piece into a numeric representation of its meaning. This lets the system compare a client request with the notes by meaning, not only by exact words.
Searching. Every find command ran three steps:
- Find similar text. The Knowledge Base picked the 20 text pieces closest in meaning to the client request, looking only at active engineers. A built-in reranker then re-sorted them, so the best matches came first.
- Group by person. Pieces that belong to the same engineer were merged into one profile by name.
- Decide and write. One call to an LLM (Amazon Nova Pro on AWS Bedrock) read the grouped pieces. It kept only the engineers it considered a fit and wrote the table.
One detail of this design matters later. The search works with pieces of text, not with people. Twenty pieces do not mean twenty engineers, because a strong profile can take several slots. When we asked for a shortlist of 20, the assistant returned 5.6 people on average. The LLM could only choose from the engineers those pieces belonged to.
The challenge: why our RAG search results weren’t enough
A shortlist can look relevant and still contain the wrong people. Most client requests combined several must-haves: a particular stack, seniority, and English level. Mentioning the right technologies was not enough. An engineer had to meet all of them together.
Our pool is mostly Ruby on Rails engineers, so Rails alone doesn’t separate them. The second requirement does, and engineers strong in both stacks are few. One request asked for a senior Rails engineer with strong React and English B2. Seven of our 111 engineers fit it, and none appeared in the top five.
The assistant’s first two suggestions were:
- A JavaScript full-stack engineer with no Ruby experience.
- A strong React developer whose interview notes show little recent Rails work.
Both profiles contained text similar to the request, but neither met its core requirements. Three properties of the search explain why:
- A skill mention was not proof of proficiency. A note such as “React: weak, basic hooks only” could still match a request for React experience.
- A strong match on one requirement could hide a missing one. Relevant React experience could bring a profile into the results despite missing Ruby skills.
- The model saw pieces of text, not complete profiles. Evidence that confirmed or ruled out a required skill could sit elsewhere in the interview report and never reach the selection step.
For recruiters, this meant re-checking profiles before trusting the shortlist. We needed the assistant to assess all must-haves using each candidate’s complete profile and put the strongest matches first.
The solution: Jev AI integration for candidate evaluation
We replaced the retrieval step with Jev, the decision model from TypeSafe AI, to evaluate complete candidate profiles against each client’s requirements. Jev returned probabilities and scores rather than written explanations, so we kept our existing LLM, Amazon Nova Pro, to explain the matches and highlight potential gaps.
Jev AI evaluated candidates, our code ranked them, and the LLM wrote the explanations.
How we implemented candidate evaluation
01. Built one profile per candidate
For each search, our application loaded every active engineer’s interview reports and current résumé, where available, from Amazon S3. It combined the documents by person and sent each profile with the client’s brief to TypeSafe’s API.
Every active candidate was evaluated, regardless of how closely their profile’s wording resembled the request. The evaluation no longer depended on which text fragments the search retrieved.
02. Defined two evaluation questions.
We configured Jev AI to assess whether each candidate met the requirements and how strong the match was:
| Question | Jev’s answer | How we used it |
| Acceptability: Does this candidate meet the position’s requirements? | A probability from 0 to 1 | The primary ranking signal |
| Fit: How strongly does this candidate match? | A score from 0 to 4, from “does not meet the position” to “exceptional match” | Tie-breaker when Acceptability probabilities are equal |
We also instructed Jev to prioritize technical interview reports when they conflicted with a résumé. Our engineers’ assessments took precedence over self-reported experience.
03. Ranked candidates in code
Our application sorted candidates by Acceptability probability, then Fit score, and selected the requested number.
04. Added explanations without changing the list
Amazon Nova Pro received the selected candidates’ full profiles and wrote a short fit explanation and potential concerns. Its instructions prohibited adding, removing, or reordering candidates.
Recruiters still type one command and review the results in the same Slack thread. The Slack command, document storage, and table format are unchanged, and each candidate’s Acceptability probability now appears as a percentage.
This let us introduce a new evaluation model without replacing the surrounding system. For a similar integration, start by defining what Jev should evaluate, which criteria it should apply, and how your application will use its scores.
How we tested shortlist quality
We compared both workflows using a fixed dataset of seven real client briefs and 111 engineers: 777 brief–candidate pairs. Both workflows used the same data snapshot. The comparison measured the complete pipelines: RAG retrieval followed by LLM selection versus full-profile evaluation with Jev and ranking in code.
First, we established which candidates met each brief. A separate LLM judge read every brief and complete candidate profile, following a written rubric without seeing either system’s output. It assigned one of four grades:
- Strong: every must-have evidenced, seniority at or above the required level, and most nice-to-haves present.
- Acceptable: every must-have evidenced, and the required level met.
- Partial: one must-have missing or unproven, or seniority one step below.
- Not a fit: the wrong role or stack, or two or more must-haves missing.
A person spot-checked the strong and acceptable assessments. These two grades counted as relevant matches and formed the reference set for evaluation.
We ran each workflow separately for requested shortlist sizes of five and 20, measuring:
| Metric | What it tells us |
| Precision | How many of the returned candidates were relevant? |
| Recall | How many relevant engineers from the pool appeared in the shortlist? |
| Ranking quality | Were the strongest matches near the top? |
We calculated precision and recall using Ragas, an open-source evaluation library, comparing returned candidate IDs with the reference set. Ranking quality was measured separately using nDCG, which gives more credit when stronger matches appear higher in the list.
For the comparison, we distinguished two Jev lists:
- Production list: returned the highest-ranked candidates even when their probabilities were low, so recruiters could review the closest available matches.
- Filtered evaluation list: applied an Acceptability threshold of 0.5 to the returned list, so it could contain fewer than the requested number of candidates or be empty.
The filter applied only to evaluation, not production. It let us measure the quality of candidates Jev rated as likely fits but could also produce an empty list. Empty lists were excluded from mean precision because they contained no candidates to assess.
Results: Jev AI vs RAG for AI candidate search
Jev AI produced more relevant shortlists and placed stronger matches earlier: the two improvements that mattered most for recruiters reviewing the first five names.
More suitable candidates in the shortlist
Production precision increased from 44% to 74%: a larger share of the suggested engineers met the client’s requirements.
For the filtered evaluation list, which included only candidates with an Acceptability probability of at least 0.5, precision reached 92.5% which is more than double the RAG baseline.
This gave recruiters a better starting point, with fewer unsuitable profiles competing for attention.
Stronger matches appeared first
Mean production nDCG@5 rose from 0.50 to 0.94, where 1 represents the ideal ranked shortlist for that brief.
For the Senior Ruby + React request, our RAG assistant’s first suggestion had no Ruby experience. With Jev AI, the first four candidates were all assessed as strong matches.
Top-five results: precision measures candidate relevance, recall measures coverage of suitable engineers, and nDCG measures ranking quality. Blue shows the filtered Jev list; green shows production. Values are rounded: 92.5% appears as 0.93. Filtered precision excludes one brief with an empty list.
A broader set of relevant options
Production recall increased from 36% to 56% for five candidates and from 30% to 89% for 20. Jev surfaced more of the suitable engineers already available in our pool, giving recruiters more relevant alternatives to consider.
This gives us a better way to match engineers to briefs that combine multiple skills, whether for our React Native development services or projects requiring React, Go, and AI expertise.
Seven briefs and one candidate pool are a small test, and the reference set was graded by an LLM judge with a human spot-check. They show a clear direction, but any team should confirm the results on its own data.
AI data security: how we protect the files an AI reads
Candidate data stays in our AWS account, with one exception. Interview Reports and résumés live in Amazon S3. Jev AI is the only outside service that sees them: each search sends it the full text of every active candidate. We made that choice deliberately and documented it.
Requests to TypeSafe use authenticated HTTPS, with the API key stored in AWS Secrets Manager. Shortlists are posted to Slack threads accessible to authorized users.
Every résumé is quarantined before entering the recruiting pipeline. Amazon GuardDuty scans the file for malware, format and size checks validate it, and Bedrock Guardrails screen the extracted text for potential prompt-injection attempts. Files that fail any check are deleted.
Jev AI returns only a probability and a score. The writing model has no tools, and Guardrails check the brief going in and the table coming out.
No check removes risk entirely. If your data is sensitive, review the provider’s data-handling terms first, as we did.
Best Jev AI use cases across industries
Jev AI is worth testing when a workflow needs to evaluate records against several criteria, not simply retrieve similar text. The output can help prioritize a review queue, classify a request, or rank available options.
Our recruiting results support one such application. The scenarios below are potential uses that require separate validation.
| Business area | Potential use | Business value and limits |
| Customer support | Classify requests by issue type and urgency using the message and account context. | Help urgent cases reach the right team. Missing context can affect classification, so uncertain cases need a fallback queue. |
| Insurance | Assess claim submissions for complexity and missing evidence. | Prioritize specialist review and identify information to request. A score should not determine coverage or replace a claims decision. |
| eCommerce | Evaluate return requests against store policies and flag exceptions. | Separate routine requests from cases needing attention. Fixed conditions, such as return windows, may be better handled in code. |
| Asset maintenance | Rank documented inspection findings using severity, asset importance, and operational impact. | Help specialists decide what to review first. This evaluates supplied findings; it does not establish that Jev can detect defects in imagery. |
| Logistics | Prioritize delivery exceptions using deadlines, shipment details, and reported disruptions. | Focus attention on time-sensitive cases. Reliable prioritization requires current data; Jev’s score is not a route-optimization plan. |
| Procurement | Compare supplier submissions against project requirements and flag gaps. | Give reviewers a structured starting point. Jev can assess supplied evidence, not verify that a supplier’s claims are true. |
What to consider before implementing Jev AI
- Keep straightforward rules in code. If a decision depends on an exact deadline, a numerical threshold, or a mandatory field, a deterministic check may be simpler and easier to audit.
- Treat scores as decision support, not guarantees. A probability depends on the evidence and criteria supplied. Missing information, ambiguous policies, or a different type of task can change how useful it is.
- Validate before automating. Compare Jev’s outputs with reviewed examples from your own workflow. Check errors, ranking quality, cost, and response time and define what happens when evidence is insufficient. Sensitive records also require a review of the provider’s data-handling terms.
For decisions affecting employment, insurance coverage, or access to financial services, keep qualified reviewers responsible for the outcome. The practical opportunity is to make their review queue more useful, not to remove their judgment.
Jev AI is worth testing when the challenge is evaluating evidence against several requirements, rather than applying fixed rules. In our recruiting workflow, it improved shortlist relevance and ranking. For other businesses, the question is whether it can deliver a similar, measurable improvement on their own data.
Final words
In our recruiting tool, evaluating complete candidate profiles with Jev AI improved production shortlist precision from 44% to 74% and brought stronger matches to the top. We kept the LLM for explanations, while Jev handled evaluation, and our code controlled ranking.
The broader lesson is to match the tool to the task. When a workflow needs to assess several requirements together, better retrieval or a more powerful text-generating model may not be enough. Test whether structured evaluation improves results on your data and whether the gain justifies the cost, response time, and data-handling requirements.
If you’re considering Jev AI, we can help evaluate it on your workflow, compare alternatives, and take a validated approach into production.
FAQ
What's the difference between Jev AI and an LLM?
An LLM generates text, such as an explanation, a summary, or an answer. Jev AI is a decision model that scores, classifies, and routes information against criteria you define. It returns typed values that your software can use directly, such as a probability from 0 to 1 or a category. Both can work together: Jev AI makes the decision, and an LLM explains it when a written explanation is needed.
Can Jev AI replace ChatGPT and Claude?
Jev AI isn’t a direct replacement for ChatGPT or Claude because it serves a different purpose. Rather than writing answers or holding a conversation, it evaluates information against your requirements and returns scores or categories that your software can use. If you need to rank candidates or prioritize requests, it may be worth testing, while ChatGPT or Claude can still explain the results in plain language.
Is Jev AI cheaper than an LLM?
It depends on the model and workflow. For evaluating 111 profiles, Jev AI was estimated at about $0.01 per search, compared with 0.08–1.50 for selected LLMs processing the full pool in one prompt. These estimates use October 2026 rates and exclude the additional LLM explanations in our Jev workflow. The LLM alternatives were not quality-tested.
When should I use Jev AI instead of an LLM?
Consider Jev AI when you need to evaluate records using defined criteria and rank or prioritize them. For writing, summarizing, or answering questions in text, use an LLM as Jev AI returns structured values such as scores, probabilities, and categories, not written responses. For evaluation tasks, test both approaches on your own examples before deciding.
