UltiHash

🚀🏆 Exciting news from the German Startup Cup-Symposium! We proudly congratulate Jonas Menesklou from AskUI on clinching the title in the #Software & #AI category at the grand live final hosted by PHOENIX group - Integrated Healthcare Provider in Mannheim! 🎉Congratulations to Tom Lüdersdorf (UltiHash) for his strong presentation and securing strong second place! 🙌 A big thank you to our expert jury: ➡️ Oliver Hehl (KPMG) ➡️ Dr. Andreas Nauerz (Bosch Digital) ➡️ Dr. Kay Stankov (Ainovate) ➡️ Tristan Werner (Deloitte) The symposium kicked off with an inspiring speech by Prof. Dr. Walther Ch. Zimmerli (President of the German Startup Cup), Dr. Roland Schütz (PHOENIX group - Integrated Healthcare Provider), and Dr. Gerd Große (GFFT e.V. | United Innovations), followed by Dr. Schütz’s keynote "Embracing Digitalization Together." Johanna Rauschenbach (PHOENIX group) led a discussion on "Future of Agile Enterprises" with Prof. Dr. Volker Gruhn (adesso SE), Dr. Andreas Nauerz, Dr. Roland Schütz, and Dr. Gerd Große. 🎤🗣️ Afterwards, the two thought leaders Prof. Dr. Volker Gruhn and Dr. Andreas Nauerz received the "Distinguished Innovator" award for their promotion of great innovative strength and knowledge transfer. 🏅 Ralf Pürner (Deloitte) provided practical insights with the "Automated Accounting Advisor" use case presentation. 💼🔍 Erik de Bueger (Sumo Logic) introduced Dr. Stefan Hartmann (Deloitte) who discussed "Security in Agile Enterprises" with Daniel Hofmann (PHOENIX group), Dr. Andreas Hamprecht (Deutsche Bahn), Prof. Dr. Sabine Rathmayer (Hochschule der Bayerischen Wirtschaft (HDBW) gGmbH), Christopher Ruppricht (SCHUFA Holding AG), and Erik de Bueger. 💻🔗 A heartfelt thanks to the PHOENIX group team — Florian Eder, Johanna Rauschenbach, Dr. Roland Schütz, Daniel Hofmann, Alexander Selzer, Eelco Wagensveld, Lukas Würth, Sandra Ley, and Vanessa Siepe — for hosting and organizing this incredible event. Thanks to all participants for their contributions to the German Startup Cup 2023/24. 📢 Looking ahead, we invite all startups to join the next season of the German Startup Cup! Register now: 🔗 https://lnkd.in/eNVtabFM #GFFT #UnitedInnovations #Startups #DeutscherStartupPokal #AI #Software #GFFT #Entrepreneurship #Tech #Digitalisierung #UnitedInnovations #Phoenix #Innovation #GermanStartupCup #Entrepreneurship

At the start, your data lake was great: a vast repository full of useful insights. But over time, the clear water became more murky. Everybody dumped data without a thought for quality or usability. With no clear catalog, searching for real insights was tough. Everyone had the vague feeling that there was a treasure trove under heaps of useless data - if only they could get at it! So data lakes’ great strength can also be their greatest weakness. As volume and variety grow, you risk a ‘data swamp’ - full of disorganized, low-quality data. On the flipside, there's ‘dark data’: high-value, but unused or even unknown. Luckily, none of this is inevitable. You can shield your lake from this fate with data governance, setting standards for data quality, security, and compliance - and implicitly enforce their execution - ensuring consistency across the board: - Establish a data catalog. Look for tools to add descriptions, tags, and categories, like The Apache Software Foundation's Atlas, or Alation. On Google Cloud, use Google’s Dataplex Data Catalog; for Amazon’s cloud, Amazon Web Services (AWS) Glue is a great bet. - Perform regular audits for obsolete / redundant data. Visibility into access patterns is also helpful for general security, so look into behavior analytics tools like Varonis DatAdvantage. Plus, all cloud platforms allow you to set up retention rules that can relocate data to cooler storage, or delete them permanently, as time passes. - Use roles to specify access rights and manage compliance. You can use built-in platform tools, or third-party tools like SailPoint and Okta for authentication. - Ingestion checks can prevent low-quality data from entering the lake. DataCleaner is an excellent off-the-shelf tool for tidying up data, but you can also build custom scripts of your own. Seems like a lot? For small- to medium-size companies new to data lakes, start with an out-of-the-box solution with built-in functionality - like Databricks and Delta Lake. Once you start to find the limits of these solutions and know your needs, explore building your own custom lake - and then apply these techniques. Remember: strategies like these need a dedicated team (or person) responsible for data governance. Make sure all this aligns with company objectives: without buy-in from internal stakeholders, the data lake might go underused anyway, making all your work to make data readily available pointless. With everyone on the same page, your data lake can remain a valuable asset for insights - instead of a swamp. Got great strategies for avoiding data swamps? Tell us in the comments.

After weeks of hard work, you’re taking a new build live. Then disaster strikes. Everything worked great in the cloud dev environment with test data, and with real data in staging. But in production, it’s chaos. ‘It worked great in staging!’ exclaims one engineer, bewildered. ‘What’s different now?’ Eventually, you find the culprit: a network config mismatch overlooked during manual setup. Horror stories like this were all too common before Infrastructure as Code (IaC). Maintenance was labor-intensive and error-prone, requiring much preparation and checking (with backups in case things went south). But in the last 10-15 years, the ‘shift left’ strategy has seen more engineers setting up their own infrastructure. To avoid bottlenecks during periods of high demand, IaC tools have emerged to allow less-experienced engineers to adjust infrastructure under expert DevOps guidance. There are IaC tools for a variety of tasks, but I’d like to focus on the current go-to solution: Terraform. I recommend avoiding platform-limited tools like (for example) Cloud Formation for AWS. A generic tool like Terraform offers a range of providears including all major cloud and container services like Kubernetes. This flexibility accelerates development, collaboration, and reliability - without binding you to a provider. Plus, you can work with the strong Terraform community and its open-source registry of provider modules, instead of relying on one company. Benefits of Terraform's IaC: • Precisely define infrastructure with declarative code. Providers automatically calculate resource dependencies and create and destroy them in a reliable order - eliminating 'it worked on the other environment' syndrome. • Use a state file as a single source of truth. Configure the remote storage (e.g. S3) to enable collaboration and prevent conflicts through version control and pull request reviews. You can even test for issues before deploying with tools like tftest. This eliminates knowledge silos, as the config is accessible to everyone (even without direct environment access). • Create modules as abstractions for multiple connected resources, so infrastructure can be clearly described in terms of structure rather than physical hardware - great for scalability. Compliant, secure modules can be written by an expert - or pulled from the Terraform Registry - and then reused everywhere, enforcing security by design. • Reduce costs by reducing manual overhead. Clarify what is running across environments, and swiftly destroy unused instances. Tools like InfraCost can even calculate the cost of changes from a pull request before deployment. It's essential, but the transition to IaC has a learning curve. Make sure to invest in training from day one, including sufficient documentation. With the right approach, switching from manual infrastructure config to automated IaC promises to save you from the horror of late-night deployment woes - and facilitate scalable, agile growth.

At UltiHash, we’re building an object storage solution that features high-performance deduplication at the object level. If you’re already using some kind of deduplication, we’d love to know more about it.

It’s 2024, and you have a mountain of data to organize - and learn from. How do you do it? Two real-life examples: In 2013, UPS upgraded their data warehouse with petabytes of structured data. This powered a project to dynamically optimize delivery routes. They analysed large amounts of logistics data to make rapid real-time route adjustments, ultimately significantly cutting shipping miles and carbon emissions. In 2021, Coca-Cola Andina leveraged AWS to build a data lake, consolidating 95% of its disparate business data and integrating analytics, AI, and machine learning. Because data was all in one place, the analytics team spent less time talking to data owners to find what they needed - increasing productivity by 80%. This fostered a culture of data-driven decision-making across the organization, as well as increasing revenue. These show the two dominant data organization patterns: data warehouses and data lakes. Here are the main differences: Data types • Data warehouses are primarily for structured data. • Data lakes can store any data type: structured, semi-structured or unstructured. Flexibility • Data warehouses require setting up a data schema upfront. This streamlines querying, but limits the ability to pivot to new data or use cases that don't fit the original plan. • Data lakes let you ingest raw data from diverse sources without prior organization - no matter its type or structure - and decide how to use it later. This approach is highly flexible but can increase complexity. Scaling cost • Data warehouses are intended for smaller amounts of operational data, and tend to require upfront investment, especially on-prem. Because storage and compute are coupled, costs tend to increase with scale - and past a certain size, large datasets become prohibitively expensive. • Data lakes tend to be more cost-effective off the bat, especially with pay-as-you-go cloud offerings. As storage is inexpensive and decoupled from compute, they can leverage serverless elasticity to scale up and down automatically, and store operational and archive data in one place. Use cases • Data warehouses are great for high-speed querying and reporting on structured datasets - ideal for critical decision-making with tools like PowerBI and Tableau. Generally a smaller group of business professional users. • Data lakes are best for vast amounts of diverse raw data, from CSV files to multimedia. This breadth allows for exploratory analysis, predictive modeling, statistical analysis, and ML. Variety of users, from analysts to data scientists. The choice depends on what kind of problems you want to solve. If you're after fast analysis within a predefined structure, data warehouses could be your go-to. On the flipside, data lakes offer flexible insights across a broad range of data, more easily scalable to large datasets. For those who’ve faced this choice before - what made you pitch for one or the other?

From traditional batch processing to real-time streaming, how data enters your platform can make a big difference to agility and decision-making. As we build a product that can integrate optimally with a range of use cases, we want to learn more about your strategies for data ingestion.

When it comes to cloud storage, hidden costs can significantly impact the total cost of ownership, catching many by surprise after they've committed. 💸 Sometimes there are costs associated with number of read/write operations and API requests; sometimes it’s about the expense of support, monitoring, or security. Even before the systems are set up, there can be hidden costs for data migration, configuration, and integration with existing tools. Whatever your hidden costs are, we want to learn about it. Let's shed some light on the real cost of cloud computing and discuss how to anticipate and manage these expenses effectively. 👇

From maintenance, to supply chains, to self driving cars, big data is transforming the automotive industry. Find out how 👇 https://lnkd.in/eXXgXS5f

Everyone working with data wants to make their work smoother and more efficient, but we want to know your biggest everyday pain points around storage. Is it keeping up with data growth? Balancing cost with performance? Making sure everything works seamlessly with the tools you already rely on? Maybe it’s something else - let us know.

Data orchestration tools are crucial for coordinating the data flow through various stages: ingestion, cleaning and transforming into insights. It is especially critical is to be able to do this reliably, with the option to automatically scale, monitor and alert in case of issues. We’ve been talking to customers using a variety of different tools to achieve this, such as Apache Airflow, Luigi, Flyte and Prefect. What do you use to orchestrate your data or ML pipelines? Let us know in the comments 👇