- Wikipedia: "An interdisciplinary field that uses scientific methods, processes, algorithms and systems to extract knowledge and insights from data in various forms, both structured and unstructured"
- There are some who say that data science is just statistics.
- Define algorithm: A set of steps to solve a problem in a finite amount of time.
- Define data: a set of values.
- Define information is data that has been processed, organized, structured or presented to make them meaningful or useful.
- More on these later.
- What did you get out of the articles?
- techtarget:
- Study of where data comes from? (I don't like this as much )
- how it can be turned into a valuable resource. (Full stop, not for business and IT strategies)
- Identify patterns--- Yes.
- Mathematics, Statistics, Computer Science, as well as Domain Knowledge.
- Tasks
- Machine Learning: DSCI 460
- Data Mining: Perhaps the search for information in data.
- I have data, I don't know what is in it so let me play with it and see what I can find.
- Discovering patterns in large data sets
- Exploratory data analysis
- Knowledge Discovery
- Cluster Analysis
- With the UFO data set
- I eliminated everything over a day.
- And everything less than half a day
- I plotted duration vs hour.
- And adjusted the axis.
-
- With the UFO data set
- Visualization
- What is "data driven decision making"?
- Process:
- Collect data
- Process, and possibly structure the data
- Interpret and summarize data so it is useful
- KDNuggets
- First note this is a HUGE "hooray for data science" web page.
- "Data science is the discipline of making data useful"
- Statistics, Machine Learning, Data-mining/analytics
- He seems to break it into three parts
- Data mining or what potential information is here.
- Statistical inference or did I really discover something
- Machine Learning, OK I discovered something, can I make predictions?
- Warden:
- What is a science?
- As a Computer Scientist I have grown up with "Anything that needs science in the name is not a real science"
- Look at merriam-webster.com
- Science is "It was chosen to promote a disciplined approach to reasoning that relied on data rather than the poorly-supported logical deductions many contemporaries favored.".
- "Data and the tools to process it are suddenly abundant and cheap. Thousands of people are exploiting this change, making things that would have been impossible or impractical before now, using a whole new set of techniques."
- While this is an old article, I like it because it shows how a new field struggles with what it will be called and what it is.
- What is a science?
- techtarget:
- A venn diagram from 2010
-
- By Drew Conway
- How do we read this?
-
- There are many of these, this one amuses me
-
- I personally have no doubt that data science
- Involves electronic manipulation of larger datasets.
- Involves mathematical/statistical knowledge to validate the results.
- Involves domain knowledge to apply the results to answer questions and make recommendations.
- Clearly communicate the results and recommendations.
- But it is not about:
- The computations
- The size of the dataset.
- The statistics
- A last thought here
- While you may never be a "Data Scientist", you may be a data scientist.
- The ability to work with a larger dataset will provide a powerful tool
- You will be able to turn data into information
- While you may never be a "Data Scientist", you may be a data scientist.
- New Note 2020: I have had several people report back that they are applying the work from these classes (101 and 201) when/where they never expected to.
- I constantly find myself looking at data differently now.
- Data is a set of values.
- Merriam Webster says it is
- factual information (such as measurements or statistics) used as a basis for reasoning, discussion, or calculation
- information in digital form that can be transmitted or processed
- information output by a sensing device or organ that includes both useful and irrelevant or redundant information and must be processed to be meaningful
- I would probably go with more of the third definition.
- The first doesn't work well for me.
- Herzog says : Any computerized file that uses columns and rows ()a tabular structure) to organize information that's represented as text, numbers and data.
- I don't like this at all, it is too restrictive.
- Data does not have to be in a digital format
- Take a look at Ancient data, modern math and the hunt for 11 lost cities of the Bronze Age
- The paper is here
- Apparently they analyzed 12,000 cuneiform texts.
-
Our first contribution is to extract systematic information on commercial linkages between cities from ancient texts. To do so, we leverage the fact that the ancient records we study can be transcribed into the Latin alphabet and digitized. We parse these texts and automatically isolate all tablets that jointly mention at least two cities. We systematically read those texts, which requires an intimate knowledge of the cuneiform script and Old Assyrian dialect of the ancient Akkadian language in which the records are written. Taking individual source context into account, this analysis identifies exclusively a subset of records that explicitly refer to trades between cities.
-
- There were 26 cities mentioned.
- The location of 15 were known.
- 11 were unknown.
- They developed an algorithm which predicted the location of cities based on frequency of trade.
-
Our second contribution is to estimate a structural gravity model of ancient trade. We build a simple Ricardian model of trade. Our model makes predictions on the number of transactions between city pairs, which is observed in our data. Further imposing that bilateral trade frictions can be summarized by a power function of geographic distance, the model can be estimated solely on bilateral trade flows and the geographic location of at least some known cities. We estimate a distance elasticity of trade in the Bronze Age equal to 1.9, surprisingly close to modern estimates.
Our third contribution is to use the structural gravity model to estimate the geographic location of lost cities. ...
-
- For test cases (cities with known locations) they are right on 2 out of three cities.
- For the unknown cities, they compared the results to the predictions of two historian's suspected location of the cities.
- Four are at or very near (50Km) suspected location
- Two match one historian's predictions but not the second.
- Four are "very far (more than 100Km) from the historians conjecture"
-
- The historians disagree on the location of Durhumit
- The model favors Barjamovic's prediction.
- All three agree on the location of Sinahuttum.
- I would love to read more about this, but time ...
- Nor does it have to be structured (See below)
- Merriam Webster says it is
- I think I will go with a set of values.
