How do you ensure your data is ready for AI?

Most of the clients I have seen failing to realise AI and advanced analytics initiatives is due to poor data quality. Also a fundamental of data management principles is that the size and scale of data quality issues need to be realised at foundational master data domains by applying detailed business validation rules across domains like Completeness, accuracy, validity, duplicity, inactivity, etc. A weighted avg of these measures will reflect that DQ index for that master data domain( customer, product, etc). A weighed average of the respective DQ indexes will indicate data health index. Measuring data quality and cleansing the date is one of the critical thing for Data centric or model centric AI model development and keep data clean

There is a significant difference between data quality for a single experiment and data quality for the enterprise. For an experiment, you can clean the a small set of data, but this doesn't solve the quality problem except for this one experiment, nor does it keep it clean. At an Enterprise level, data quality starts with governance and accountability. With Data Products, the governance/ownership can be more distributed, but you need to fix the data at the source to be able to scale AI, and not have data integrity issues.

To make your data AI-ready, focus on five key steps. First, assess and improve data quality to avoid unreliable AI outcomes. Use specialized tools for data cleansing and validation. Second, integrate data from different sources into a single repository to enhance its usefulness and remove silos. Third, keep your data clean and consistent by eliminating errors and duplicates. Fourth, securely store your data in a readily accessible cloud solution like Amazon S3 or Azure Blob Storage. Finally, you can use historical data to train your AI models for better pattern recognition and predictions. These steps ensure accurate and reliable AI performance.

Data quality is the bedrock of AI efficiency. But what does 'quality' mean in the context of AI? It's not just about clean data; it’s about relevant, consistent, and accurate data that mirrors the real-world scenarios AI is expected to handle. Example: Consider a retail giant using AI for customer recommendations. If the data is outdated or inconsistent, AI could suggest winter coats to a customer in the peak of summer, leading to lost trust and sales. Actionable Steps: Implement robust data cleaning processes, ensure data consistency, and establish a continuous data quality assessment mechanism.

High quality outcomes do not just come from data they comes from knowledge and people. Decision Intelligence is the science of better decision-making. The saturated mindset that all AI must start with data is fading with this resurgence of symbolic and semantic technologies. Increasingly, the future looks neurosymbolic, incorporating knowledge graphs and powerful reasoning engines to enable outcomes to be accurate and auditable, even in the face of poor quality of missing data. Anything that can be done by humans can do at scale with technology, but it takes a composite and hybrid approach. That now includes complex human decision-making.

Building an analytics layer with clean data through data integration is vital for the success of any AI project. The typical steps in creating the analytics layer are 1) Load the raw data from disparate sources into a data lake or data warehouse. There are a lot of integration tools that can facilitate the same. The optimal way is to merge data that has been changed (CDC). 2) Create derived data through necessary transformations, data cleansing, data formatting, and business rule implementation. 3) Build the analytics layer (tables, dimensional models, materialized views, .. etc.) from the derived data. Data scientists, ML engineers, and AI engineers use this layer for the development of models.

In the context of data integration, understanding different data storage architectures is crucial: Data Warehouse: A structured repository optimized for analytics. It pre-processes and categorizes data before storage, making data retrieval fast. Data Lake: A vast storage system that holds raw data in its native form, including structured, semi-structured, and unstructured data. It's flexible but may require more processing during data retrieval. Data Lakehouse: A hybrid combining the best of both worlds. It offers the structured organization of warehouses while retaining the vast raw data storage of lakes. Selecting the right architecture aids seamless data integration.

AI thrives on integrated data - a harmonious blend of information from varied sources. The challenge? Ensuring this integration is seamless and meaningful. Example: A healthcare provider integrating patient data from various systems - EMR, wearable devices, and patient surveys - to predict patient health risks more accurately. Actionable Steps: Adopt modern data processing and open data storage tools , technologies and strategies for effective data integration, such as Spark, Delta and similar distributed processes, and ensure data compatibility and interoperability.

Most data storage is PHP or SQL these days. PHP: Strengths: PHP is a server-side scripting language, allowing dynamic data presentation. It can handle file operations and communicate with databases, facilitating custom labeling tools. Weaknesses: Not inherently designed for data operations. Might be slower and less efficient for large-scale data labeling tasks. SQL: Strengths: SQL is a specialized language for managing and querying databases. It can quickly and efficiently label, update, or retrieve large datasets. Weaknesses: Requires structured data in databases. Less flexible for creating dynamic, interactive labeling interfaces compared to PHP.

In the world of AI, data labeling is akin to teaching a child to understand and categorize the world. It’s about making the data understandable and actionable for AI algorithms. Example: An autonomous vehicle company using meticulously labeled data to distinguish between a pedestrian and a street sign. Actionable Steps: Invest in accurate and comprehensive data labeling, considering both automated and manual labeling strategies.

Data augmentation is a key part of the expansive and generative knowledge that AI can offer. The key part here is for organizations to tap into their unreserved wells of information in their unstructured data. Herein lies the knowledge of the company’s products and processes that is often undocumented. By transforming unstructured information into structured and then linking it to structured data, companies can extend the datasets.

We use MS for this, but there are many tools. PowerBI provides a robust suite of tools for data transformation. Within its Power Query Editor, users can perform operations like filtering, sorting, and adding calculated columns. More advanced transformations like pivoting, merging queries, and conditional operations can enhance data visualization and analysis. These features in PowerBI support the data augmentation process, enabling users to derive more insights from existing datasets, ensuring the data's relevancy and breadth for AI applications.

From an EU perspective the usage of any personal data for AI training is never a good idea. I do see a lot of people advocating using "consent" as the basis, however what if the consent is withdrawn? It is impossible to remove data from an AI once "entered", and hence your only choice is to scrap your AI and retrain without the data you cannot utilize any longer. In my view the solution is to use synthetic or anonymized data for training. Remember in training the "patterns" in the data is the key. You can train an AI to recognize a cat using only synthetic pictures - why cant you do the same, when it comes to data sets containing personal data? As long as you cannot see if the data is real or "synthetic" - you will get the same result.

As part of your data governance program, all your sensitive and confidential data should be automatically classified. This provides the proper labeling to ensure that private data on your customers and employees are utilized according to the proper purpose of collection and it can be deleted and destroyed upon the data retention schedule. Different global and industry regulations can also apply to adhere to privacy compliance regulations as well.

Data governance is the framework that ensures data is managed appropriately and ethically, especially crucial in the AI landscape. Example: A financial institution using AI for credit scoring must govern its data to ensure compliance with regulations like GDPR and avoid biases. Actionable Steps: Establish clear data governance policies, focusing on compliance, security, and ethical use of data.

Data Governance can ensure your data is ready for AI and managed responsibly and efficiently throughout its lifecycle. Effective data governance is about having a structured approach to managing your data, with clear policies, dedicated roles, measurable metrics, and tools aligned with your overarching AI objectives.

We use MS Purview for governance. It automates data discovery across platforms, ensuring AI projects utilize clean, compliant data. With visual data lineage, we can trace data origins, enhancing transparency. Classifiers identify sensitive data like PII, ensuring AI models don't inadvertently breach privacy. By facilitating stakeholder collaboration, defining access controls, and encrypting data, governanceensures AI's power doesn't compromise data privacy.

Before diving into AI programs, laying a solid foundation of Data Governance is non-negotiable. This safeguards your data's quality, security, and ethics. Key DG components like data quality, access controls, and ethical guidelines seamlessly align with AI's responsible usage. Data Governance acts as the guardian, ensuring AI operates within ethical boundaries, securing valuable insights, and fostering trust. It's the essential bridge between data and AI, ensuring they work harmoniously.

Data governance is the key aspect for AI, before the model development. Data is the heart of AI and the outcome will directly depend on the training data used. Ensuring data quality by data labelling, data augmentation and data anonymization help create an accurate model. Synthetic data creation to avoid Bias is also one of the key measures which is often missed out.

Developing a data strategy is also vital in ensuring your data is ready for AI. A data strategy outlines how your organization will collect, store, manage, and use data to achieve its goals. It includes a clear plan for data governance, quality, security, and privacy. It should align with the organization's overall business strategy and be flexible enough to adapt to changing business needs and technology.