You're working on a complex data set. What's the best way to approach it?

Antes de qualquer análise é importante entender o contexto. Para isso, é necessário um trabalho de comunicação com as pessoas interessadas pelo resultado da análise e com as pessoas que transacionam as informações e trabalham gerando esses dados. Muitas vezes, a maior parte do trabalho será "mergulhar" no processo para entender as regras de negócio e somente depois começar um trabalho de exploração dos dados.

It is important first to understand the structure of the data, the variables, the research questions or objectives, what we want to analyze and who are the audience we are working with and what type of data if it is a questionnaire or time series data for example

Complex or messy data is very common. Leaders may have experienced based guesses that are often incorrect. Understand questions of interest. Many times the information available may not be able to prove the point. Always remember the best approach is to keep an open mind to find overlooked insights based on sound methods that will drive a better business outcome

Don't jump straight into the data! Ask why you're analyzing it, who needs the answers, and where the data comes from. This helps you focus on what matters and avoid getting lost in the details.

In a CRO, we handle critical research data, collecting, cleaning, analyzing, & utilizing it to extract endpoints. Given the immense data stream and associated risks, we tend to collect data at every step to ensure accuracy and avoid discrepancies. However, we often overlook the importance of determining the usefulness of the additional variables collected and cleaned. This oversight adds unnecessary work to data collection, qc & maintenance processes. It is crucial to maintain focus on the endpoint analysis and align the variables with the study needs and context. This can be done by regularly assessing the relevance of collected variables, and maintaining good communication & collaboration between different departments handling the data.

Exploring data is vital for the following reasons: - uncovering errors (data type, values, etc) - discovering relationships between merged and unmerged tables. - identifying trends, inaccurate data, or ways to display data in a storyboard way The point being, you’ll never understand the way the data is supposed to be told if you don’t get your hands dirty to explore it. Whether it’s fixing models, building reports, or discovering trends, the more you investigate the data, the more understanding you begin to build in your coding and data interpretation skills.

Exploring the data is the process after data is collected or coming from online sources, you must summarize your data by descriptive statistics tables to find if there any missing values, detecting outliers that might affect on your distribution of the data,you can check the skewness and kurtosis in this case, also you can explore the data by creating charts and trends to find some preliminary relationships for your further analysis

Each data analysis tool or data visualization tool, Python or R can provide you some features and functions to explore data, for example: Power Query with data profiling tools, Python with Pandas has describe(), info()... It is a complex data set so be patient exploring data

Acredito que nesta parte cabe também diferenciar o que são dados mais variáveis dos que se mantém. Ajuda bastante para a análise posterior com previsões

Exploring the data will help us determine what is it in that data that can be crucial for us to solve a business problem, once we know in and out of the data, then only we can determine it's proper usage.

Sometimes you need to change your variables by transforming some continuous variables into categorical variables in case your analysis is related to a specific topic, also it can be to standardize your data due to violation of normality, you can also use the factor analysis and dimension reduction in case you are dealing with Likert scale variables to reduce your items to less variables or factors

I agree with the above and I would also like to add that in my experience it’s an iterative process and iteration over the steps will allow you understand the source of the error or anomalies (good or bad) in the data.

Normalmente un buen input es más importante que el modelo en si. Comó dice el dicho, “garbage in, garbage out”. Si el modelo recibe basura no devolverá más que basura. Por eso muchas veces la clave del éxito es un buen trataniento previo de los datos, tomando decisiones sobre como lidiar con los valores nulos y atípicos y aplicando una ingeniería de características extensiva, poniendo todo el conocimiento del dominio al servicio de esta última.

Mi gran amigo Excel, es una herramienta poderosísima para separar contenido de los datos ordenarlos o agruparlos, puedes usar tablas y resumirlas en tablas dinámicas, agrupar y crear atributos nuevos para hacer reportes a medida, luego que sabes como debe quedar tu base de datos, es muy importante, crear una macro para que se automatice el proceso de orden, modificación y limpieza de los datos, notaras que hay información menos relevante que es posible omitir y otra que es necesario detallar. La practica hace al maestro y los errores son solo aprendizaje.

After exploring data, you might have some ideas what data fields you might need, then starting for preprocess them: change datetime format, create calculated fields. Image the dataset is your house and now you need to clean and organize your house for "living" in it

As your data is cleaned and ready for analysis, first you have to understand what the type of variables are used in the analysis if it is categorical or continuous variables, what are your factors or independent variables and the dependent variables, you can use several statistical tests like ANOVA, Regression, logistic regression, mixed model linear analysis, chi square, repeated measures ANOVA, noted depending on the objective you can use each of them or more, a wide of statistics software can be used like SPSS, R ,Stata or Python depending on your budget, and your knowledge

Realize uma análise exploratória detalhada para entender as relações entre as variáveis, identificar padrões e insights importantes e validar suposições.

Cuando ya estas contento con la base de datos resultante del procesamiento de datos, PowerBi. es a mi juicio la herramienta mas simple para análisis de datos y crear tableros de datos atractivos y que además te permite crear campos calculados y generar medidas sobre las tablas de tu base de datos para llegar a generar el reporte grafico que desees para que los interesados.

Importante além dos dados que você coletou, ter também dados não necessariamente da empresa, mas sim do produto, do período e também do modelo, fatores que nem sempre ficam claros apenas em planilhas e trazem muitas variáveis

It is important after analysing the data to communicate and explain your statistical findings to your audience so they can write the conclusion, limitations and recommendations

Visualize the data Once you have collected, cleaned and analyzed the data, you can use a method like data visualization to summarize the information and ensure it's meaningful. Visualization tools can help you understand the data and present it to others in an engaging manner. Colorful charts and graphs can represent hundreds of points that demonstrate a trend or a powerful narrative. A sales data run tied to a geographic region, for example, may help you decide to reallocate marketing funds.

Abordar datos complejos siempre requiere un enfoque estructurado. Por supuesto algunas mejores prácticas incluyen la limpieza y preprocesamiento de datos, la exploración y visualización para comprender su estructura, la selección de modelos adecuados y técnicas de análisis estadístico. Es necesario hacer una validación rigurosa de los resultados. Además, el uso de herramientas como machine learning y técnicas de minería de datos puede ayudar a extraer información útil.

Um passo importante que gosto de dar quando faço as análises é entender melhor como outros setores da empresa estão impactando também no resultado do produto, que não necessariamente aparecem em relatórios. Quais produtos estão nos criativos sendo usados em Ads, quais o marketing tirou mais fotos e teremos mais banners, quais estamos com matéria prima escassa, são muitos fatores que impactam nas decisões a serem tomadas