Show HN: Nomadic – Minimize RAG Hallucinations with 1 Hyperparameter Experiment https://ift.tt/CU4TJ6V

Show HN: Nomadic – Minimize RAG Hallucinations with 1 Hyperparameter Experiment Hey HN! Mustafa, Lizzie, and Varun here from NomadicML ( https://nomadicml.com ). We’re excited to show you Nomadic ( https://ift.tt/fzpFHdU ): a platform focused on parameter search to continuously optimize AI systems. Here’s a simple demo notebook where you get the best-performing, statistically significant configurations for your RAG — and improve hallucination metrics by 4X in just 5 minutes — with a single Nomadic experiment: https://ift.tt/u0dOvYl Our lightweight library is now live on PyPI (`pip install nomadic`). Try one of the README examples :) Input your model, define an evaluation metric, specify the dataset, and choose which parameters to test. Nomadic emerged from our frustration with existing HPO (hyperparameter optimization) solutions. We heard over and over that for the sake of deploying fast, folks resort to setting HPs through a single, expensive grid search or better yet, intuition-based “vibes”. From fine-tuning to inference, small tweaks to HPs can have a huge impact on performance. We wanted a tool to make that “drunken wander” systematic, quick, and interpretable. So we started building Nomadic - our goal is to create the best parameter search platform out there for your ML systems to keep your hyperparameters, prompts, and all aspects of your AI system production-grade. We started aggregating top parameter search techniques from popular tools and research (Bayesian Optimizations, cost-frugal flavors). Among us: Built Lyft’s driver earnings platform, automated Snowflake’s just-in-time compute resource allocation, became a finalist for the INFORMS Wagner Prize (top prize in industrial optimization), and developed a fintech fraud screening system for half a million consumers. You might say we love optimization. If you’re building AI agents / applications across LLM safety, fintech, support, or especially compound AI systems (multiple components > monolithic models), and want to deeply understand your ML system’s best levers to boost performance as it scales - get in touch. Nomadic is being actively developed. Up next: Supporting text-to-SQL pipelines (TAG) and a Workspace UI (preview it at https://ift.tt/awOubry ). We’re eager to hear honest feedback, likes, dislikes, feature requests, you name it. If you’re also a optimization junkie, we’d love for you to join our community here https://ift.tt/uZTkVhm September 5, 2024 at 09:14PM