AI Tools for Literature Reviews

This is an analysis of the effectiveness and reliability of three AI tools for conducting literature reviews: Manis, Gen Spark, and Gemini AI from Google. These tools were evaluated based on the accuracy and existence of the references they provided in literature reviews.

Evaluation Criteria

The prompt used for testing:

• Generate a structured literature review on a specified topic.
• Include key research themes, evolution, important studies, conflicting viewpoints, literature gaps, and future research suggestions.
• Use peer-reviewed sources and present the output suitable for thesis introduction.

Tool Comparison

1. Manis

• Execution Time: Approximately 3 minutes.
• Output: A 14-page document with 38 references.
• Reference Accuracy: 84% accuracy, with a 16% hallucination/failure rate (e.g., incorrect journal titles, years, or repeated references).
• Advantage: Provides a PDF file with detailed references.

2. Gen Spark

• Execution Time: 5-7 minutes.
• Output: 19 references in a less convenient format for extraction.
• Reference Accuracy: 26% hallucination/failure rate (multiple non-existent references).
• Drawback: Lower number of references and higher rate of inaccuracies.

3. Gemini AI

• Execution Time: Approximately 20 minutes.
• Output: A detailed 61-page document with 105 references, including tables and sections.
• Reference Accuracy: 1% issue, primarily involving non-peer-reviewed sources like theses and inaccessible books.
• Advantage: Offers highly detailed outputs with the most accurate references.

Conclusion

• Winner: Gemini AI with the lowest hallucination rate and detailed, comprehensive outputs despite some minor format adherence issues.
• Runner-Up: Manis, due to its balance between speed and accuracy.
• Third Place: Gen Spark, due to high rate of inaccuracies and fewer references.

Overall, Gemini AI provides the most effective tool for literature reviews with its thoroughness and low error rate. Researchers should double-check all AI-generated references for accuracy.