James Peng

The high-end graphics cards from Nvidia, such as the H100, have catalyzed the emergence of dedicated Rent-A-GPU farms, expanding beyond traditional hyperscale providers like AWS, GCP, and Azure. Initially, supply chain limitations coupled with high demand enabled these specialized farms to purchase these cards—each costing nearly $40,000—in bulk and offer them for rent. By late 2023, rates were around $8 per hour under commitments ranging from one to three years. Today, the landscape has shifted significantly. Some of the largest Rent-A-GPU providers have adapted to changing market dynamics by offering GPU spot markets with commitments as short as 48 hours, with rates as low as $1.8 per hour or even less. This suggests that the supply chain for high-end GPUs has finally caught up with demand, potentially heralding a substantial decrease in costs not only in the rental market but also in capital expenditures for owning these GPUs. For the first time, there is downward pressure on the prices of high-end GPUs. This development is particularly promising for real-world applications of Generation AI. The significant reduction in inference costs is crucial for the viability and value creation of GenAI within enterprises. It represents the first of many indicators that we are approaching the rapid realization of game-changing use cases, transforming how businesses leverage advanced AI capabilities. At KamiwazaAI we believe the start of an 5th Industrial Revolution Enterprise #5IR is around #1trillionInferences a day or in the range of ~$3b dollars a day utilizing chat GPT4 at .003 per Inference (100 tokens). A 1000x decrease in the cost of Inference is only 18 months away or so and its time NOW to begin implementing the technology to keep up!

21

5 Comments

According to Eric Schmidt, in the next year or so, the union of the large context window, agents, and text-to-action, when delivered at scale, is going to have an impact on the world that no one understands yet. Listen to the fascinating discussion in the "The Age of AI, Eric Schmidt" youtube video - https://lnkd.in/dFTKgBVY

52

1 Comment

We just released our first pretrained draft model that we use to accelerate Llama 2 7B and CodeLlama 2 7B on Ryzen AI. #AMD #AI #ML

164

1 Comment

At Elemental Cognition, we’re transforming #GenAI solutions using #ApacheKafka and Confluent. In this The New Stack article, I shared how our AI platform uses Kafka and Confluent Cloud to stream and vet data in real-time before the EC Reasoning Engine applies formal logic for transparent, accurate solutions. Check out the article for a deep dive into the best practices we’ve learned for building reliable GenAI apps. https://lnkd.in/eU5RupZu

41

4 Comments

Is your team in the top 1% of AI adoption? If you've finally got a model personalized to your use case that is performing well, you're probably in the top 1%. How are you deploying that model so that it can start powering your business? If you're still pulling together your strategy, check out LitServe, our next-gen serving engine. You won't regret it 🚀⚡️

14

1 Comment