Yup, yet another R texn-an package. I trace a long history on this one, starting with tm some 6 yrs ago. Have settled on tidytext for all BOW work nowadays.
Anyway, interesting pieces of this article are [1] its dataset - UN speeches etc, [2] ref to papers that show how to do unsupervised (LDA) as well as supervised learning (classification) on text, etc.
Will write more later, this is it for now.
Yup, as suspected, it does use R's well-ish-known arules package and the datasets in that package.
I have to wonder if there's any other datasets out there and available for in-class demos.
Problem with MBA typically is that at the SKU level there's way too many items to handle and at the category level, it just isn't as interesting only.
Some sorta hierarchical setup might help perhaps, eh?
This blog closed after an unfoirtunate run-in with Batch 8, CBA. I'd then decided to close this blog for CBA course-related work.
But nothing stops me from restarting it for keeping track of my own work. Which is what follows.
I read many posts online on tech and code. This blog seems like a good place to keep track of some of the more interesting or useful ones.
+++++++++++
First, here's a nice post on how to use LASSO (Least Absolute Shrinkage and Selection Operator) to do [political] micro-targeting and online-ad-retargeting using FB 'Likes' and preference data on the one side and the Big-5 personality framework on the other.
Let me quote from the blog itself:
Basically, Microtargeting is the prediction of psychological profiles on the basis of social media activity and using that knowledge to address different personality types with customized ads. Microtargeting is not only used in the political arena but of course also in Marketing and Customer Relationship Management (CRM).
So, what's *LASSO*? An extension of the classical linear regression wherein, given many IVs on the RHS, we fish for the 'best'/'most important' ones by *shrinking* all coefficients to zero via a constrained optimization routine neatly packaged and delivered via the **glm** R library.
Another advantage of this post - it provides dummy data to test the code on. As well as decent interpretations for the shrunk yet non-zero beta coefficients in the output.
Even better, since its all pure-R, *shiny-fying* it for later use in a PGP classroom is eased.
Note: Machine Learning Basics for Marketers (MLBM) is on offer for the first time in Term 7 this acad year.
+++++++++++
Well, will close here for now. More posts will hopefully follow and this attempt won't fizzle out yet again.
Sudhir Voleti, Oct 2019.
Your Group Homework is coming up. I describe your Task below, pay careful attention to detail, pls.
Its actually two tasks which are only vaguely related. Past feedback says its better to combine all homeworks into one to enable groups to better plan and schedule work.
Task 1: Web scraping
Step 1: Go here and examine the page.
Step 2: Scrape the page and tabulate the output into a data frame with columns 'name, url, count, style'.
Step 3: Filter the data frame. Retain only those beers that got over 500 reviews. Let's call this Table 1.
Step 4: Now for each of the remaining beers, go to the beer's own webpage on the ratebeer site, and scrape the following information:
'Brewed by, Weighted Avg, Seasonal, Est.Calories, ABV, commercial description' from the top of the page.
Add these fields to Table 1 in that beer's row.
Step 5: Now build a separate Table for each beer in Table 1 from that beer's ratebeer webpage. Scrape the first 3 pages of reviews of that beer and in each review, scrape the following info:
'rating, aroma, appearance, taste, palate, overall, review (text), location (of the reviewer), date of the review.'
Store the output in a dataframe, let's call it Table 2.
Step 6: Create a list (let's call it List 1) with as many elements as there are rows in Table 1. For the i_th beer in Table 1, store Table 2 as the i_th element List 2.
Task 2: DC from Twitter API Step 1: Read up on how to use the twitter API here. If required, make a twitter ID (if you don't already have one).
R has a dedicated package twitteR (note capital R in the end). For additional functions refer twitteR.pdf (twitteR package manual).
Step 2: Recall three evaluation dimensions for beer at ratebeer.com, viz. aroma, taste and palate. More than the dictionary meanings of these words, its how they're used in context that's interesting.
So, pull 50 tweets each that contain these terms.
Step 3: Read through these tweets and note what product categories they typically refer to. Insights here could, for instance, be useful in designing promotional campaigns for the beers. We'll do text analysis etc next visit.
Deliverables: There are 4 deliverables per group - [1] R markdown of your (well-documented) code and a storified narrative of your thinking behind the code.
[2] Table 1 as an csv file. List 1 as an .RData file.
[3] List of tweets you scraped in a .csv file.
[4] List of brands and product categories that those terms were used in context of.
Zip these deliverables and submit into the drop box before deadline.
Deadline: Midnight before your Visit 2 starts.
Any Qs etc, ask Sudha or Aashish in the Tutorial.
Sudhir
There will be three homeworks in DC - two individual and one group.
This is your second individual homework and involves relatively low-tech DC on individuals' locations. Follow the instructions below step by step.
Assignment Instructions:
1. Form a Whatsapp group with your group mates. 2. Whenever you travel or visit different places as part of your everyday work, share your location to the Whatsapp group.
For example - if you are visiting an ATM, your office, a grocery store, the local mall etc., then send the WhatsApp group a message saying: "ATM, [share of location here]."
Ideally, you should share a handful of locations every day. Do this DC exercise for a week. Its possible you may repeat-share certain locations.
P.S. I'll assume you have a smartphone with google maps enabled on it to share locations with.
3. Once this exercise is completed export the WhatsApp chat history of DC group to a text file. To do this, see below:
Go to WhatsApp > Settings > Chat history > Email Chat > Select the chat you want to export.
4. Your data file should look like in this:
28/02/17, 7:17 pm - aashish pandey: location: https://maps.google.com/?q=17.463869,78.367403
28/02/17, 7:17 pm - aashish pandey: ATM
P.S. As you can see, if you have any queries or issues running this thing, reach out to Pandey ji first.
5. Now compile this data in a tabular format. Your data should have these columns -
a - Sender Name
b - Time
c - Latitude
d - Longitude
e - Type of place
6. Now comes the deliverable part. Remember this is an individual assignment even though you need your group's help to form and export the WhatsApp chat history.
Extract your locations from the chat history table and plot it on google maps. You can use the Spatial DC code we used on this list of latitude and longitude co-ordinates or use leaflet() package in R to do the same.
Remember: you should extract and map only your own locations, not that of your team-mates. They will do for theirs.
7. Deliverable: A Markdown file that shows the code you used to plot your locations over a week.
Analyze your own movements over a week *AND* record your observations about your travels as a story that connects these locations together.
8. Deadline: 19-March-2017 (Sunday) midnight. Sudha will make a dropbox where you can share the markdown documents as html files.
9. Submission Format: Individual submissions only. Name your html file as 'your-ISB-ID-number DC HW 2.htm' Inside the html file, introduce yourself with name, city, ISB ID number etc in the first paragraph.
That's it for this simple exercise, folks. Any Queries etc, feel free to contact me.
Sudhir
This post is to put up some good practices and (optional) readings for DC. More such posts with readings may follow in the days to come.
Good practices:
1. If you haven't already you should register and signup for daily email newsletters from r-bloggers (for creative uses of R code and info on new packages). Even though 80% of the posts on the daily email may not be of direct interest to me, the other 20% makes it very worthwhile indeed.
For example, here's a post detailing job market trends globally in Data science jobs (with data and visualizations done in R).
2. Similarly, signup for programmable web (for API related directory and news) as well.
3. www.kaggle.com hosts data science competitions, releases datasets and tutorials. Can use your FB or goog login. We'll be using some kaggle datasets in NLP. Good practice to keep checking what's new and hot on kaggle from time to time.
4. Replicate *all* the classwork code line by line. Especially so if you're new to R. Lookout for new functions that may come in (do ?function_name in console to see its description), read inline comment documentation carefully, etc.
Should you have trouble running any particular piece of code, search the web, ask peers etc. The coming DC tutorial on 4-March which will be conducted jointly by Sudha and Aashish Pandey is another good opportunity to get clarifications.
Some (optional) readings:
1. This is the NYT article on 'Data Janitor' work we saw in Session 5.
2. This is the NYT article on 'the data driven life' about the quantified self movement.
3. The next couple of readings relate technology to data collection and data use (from the Economist): 'Getting to know you', is about the various ways in which data is collected about consumers online.
4. 'The world wild web', extrapolates some of what we are seeing into the future and asks 'Where are we going?'.
5. From India, here's an article from ET on how data brokers are syndicating and selling user data sans oversight. 'How data brokers are selling all your personal info for less than a rupee to whoever wants it'.
6. Recall the Xerox-evolv caselet we did in class - The one where psychographic Likerts were combined with straightforward machine learning? We then discussed implications, pros and cons etc, and speculated on when such practices may spread worldwide and into India. Well, speaking of India...
'Startups, and India Inc use psychometric tests to peek into potential recruits’ minds'.
Update: Some more readings of interest
7. Recall the 'Exponential learning curve' example in session 5? Well, here's an article I wrote for the NASSCOM sales and Marketing community I wrote a few months ago on that issue.
8. Recall the GE example in 'Information Imperative' in Session 5. Here's the interview by McKinsey of GE's CEO Jeff Immelt on that topic from Oct 2015. (Might require you to register for free with McKinsey quarterly).
9. Recall what the GEexample led to? A discussion on predictive analytics and maintenance on the human machine.... Here's a timely piece from the Economist from 2 days ago on the data revolution in personal healthcare 'A digital revolution in health care is speeding up'.
Well, that's it for now. Watch this space for more to come.
Sudhir
A series of homeworks are coming your way. The first is described below. I will putup details for the others shortly.
Pls watch this ~ 20 minute video carefully. It features Scott McDonald of Condé Nast holding fort on where Marketing Research is headed in the next decade. Slightly dated but still very relevant.
“Social Technological and Economic forces affecting Marketing Research over the next decade”
Now, for your HW, pls answer a few simple Qs (True-False, fill in the blanks variety) about the above talk in the following survey:
HW Notes:
(i) This is an individual-only HW. Since it involves no R, consulting peers is not permitted.
(ii) I found that using earphones works great in making out what the speaker is saying much more clearly than ordinary speakers. FYI.
(iii) Deadline: The HW should be completed and submitted latest by midnight 10-March Friday.
Any Qs etc, pls feel free to email me or use the comments section below.
Sudhir Voleti
Reg your TABA exam pattern, here are some details. I just finished making the exam.
Any Qs etc, ask Pandey ji, me or at the comments section below.
Sudhir
I got some queries on additional readings. I'd also resolved to upload some readings to the blog.
Pls find the same below. I'm quoting from an email response I'd sent recently.
First look at the working paper I'd uploaded onto LMS. It has further academic references in its bibliography section. This is for both topic modeling and matrix factorization.
In addition to the above the following could be of use:
========
Topic modeling:
Wikipedia entry on topic models
Topic Modeling 101 simple example
Technical paper on Topic model estimation
========
On Matrix Factorization
A simple introduction to Matrix Factorization
Non-negative matrix factorization in linear algebra (Wiki entry)
Matrix decomposition for data mining (technical paper)
========
These readings are optional from an exam point of view.
Sudhir
Pls find below your two group homeworks. There's ample time to do them and Aashish will discuss them in his tutorial this Sunday.
=======
Task 1 – Empirical Topic Modeling from First Principles:
Recall we did factor-An from first principles by simulating a small data matrix, factorizing it and recovering the original (with errors)?
We'll repeat that simulation-analysis-recovery exercise, only this time with topic mining instead.
Follow the steps below and lookup sample / example code etc in the homeworks folder in LMS.
Step 1 - Choose 3 completely different subjects. E.g., I'd choose "Cricket", "Macroeconomics" and "Astronomy". You pls choose any 3 other, very different subjects / fields.
Step 2 – Scrape the first 50 websites for each of these selected subjects in google search and extract their text. Use python google search code. Thus, we'll have 3 different sets of 50 documents each, one for each subject.
Step 3 – Now create a simulated corpus of 50 documents thus: The first of the 50 documents is a simple concatenation of the first document from subject 1, from subject 2 and from subject 3. Likewise, for the other 49 documents.
Thus, our simulated corpus now has 'composite' documents, i.e. documents composed of 3 distinct subjects each.
Step 4 – Run the latent topic model code for k = 3 topics on this simulated corpus of 50 composite documents.
Step 5 – Analyse the topic model results - wordclouds, COGs, topic proportions in documents. USe the classwork code for Lift and ETA directly.
Step 6 – Comment on (i) whether the topic model is able to separate each subject from other subjects. To what extent is it able to do so?
(ii) Are there mixed tokens (with high lift in more than one topic)? Are the highest LIFT tokens and the document topic proportions (ETA scores) clear and able to identify each topic?
(iii) What are your learnings from this exercise.
Your deliverable should be a R markdown that documents your efforts and/or results from each of the above steps. Do mention which subjects you chose and present the exercise as a narrative or a story, as far as possible.
=======
Task 2 - Training a machine to classify tweets according to sentiment.
Step 1 – Choose any six different recent twitter hashtags with or without sentiment content (e.g. #ClimateChange, #Trump, #Demonetization, #Kejriwal #Technology etc)
Step 2 – Extract ~ 500 tweets for each hash tag. You may use the twitter API conector and associated R code (or equivalent in python, if you wish).
Step 3 – Stack all the ~ 3000 tweets into one corpus.
Step 4 – Remove #keywords, web URL, @user_name from tweets. Clean the raw corpus,basically.
Step 5 – Make a unique tweets corpus (~ 2500) out of ~ 3000 tweets. Drop duplicates due to retweets etc.
Step 6 – Randomly select 70% tweets (training data) and classify them manually as positive(1), neutral (0) or negative(-1).
Step 7 – From this training data, build a simple classifier model (as we did in the simple classwork exercise). Split the sample into two-thirds (calibration) and one thirds (holdout) and check the prediction accuracy of the model. Build its confusion matrix.
Step 8 – Try changing the pre-processing a few times - dropping most common and uninformative words using the stopwords filter, for instance. Does it affect prediction accuracy?
Step 8 – Using the best classifier model, classify remaining 30% tweets (virgin data).
Step 9 – Write a narrative commentary on this process - (i) what hastags you picked and why. (ii) What was the distribution of sentiment across your corpus. (iii) What was the predictive accuracy of the model and did re-processing the raw data help improve it? (iv) Learnings from the exercise.
Your deliverable is again a R markdown that documents your efforts and/or results from each of the above steps. Do present the exercise as a narrative or a story, as far as possible.
=======
Deadline is the last day of Term 2 for both assignments.
Any Qs etc, contact us. Good luck.
Sudhir
Pls find below all your individual assignments for TABA. Vivek will putup the requisite dropboxes. The assignments are due on 27th November (Sunday) midnight.
=====
Task 1 – Text-Analyzing a simple set of documents.
Imagine you're a Data Scientist / consultant for a movie studio. Your brief is to recommend the top 2-3 movie aspects or attributes the studio should focus on in making a sequel.
The aim is to get you to explore with trial-and-error different configurations of possibilities (e.g., what stop-words to use for maximum meaning? TF or IDF? etc) in the text-An of a simple corpus. You are free to use topic modeling if you wish but it is not necessary that you do so.
Step 1 – Go to IMDB and extract 100 reviews (50 positive and 50 negative) for your favourite movie. You can refer to the code provided for reviews extraction from IMDB (IMDB reviews extraction.R) on LMS.
Step 2 – Pre-process the data and create Document term Matrix. Check word-clouds and COGs under both TF and TFIDF weighing schemes for which configs appear most meaningful / informative. Iterate by updating the stop-words list etc.
Step 3 – Compare each review's polarity score with its star rating. You can choose to use a simple cor() function to check correlation between the two data columns.
Step 4 - Now, make a recommendation. What movie attributes or aspects (e.g., plot? star cast? length? etc.) worked well, which the studio should retain? Which ones didn't work well and which the studio should change?
Step 5 – Deliverable: Create a markdown document as your deliverable. Keep only the final configuration that you arrived at. In the text you can describe the trials and errors you did. Ensure the graphs/tables on which you are basing your recommendations are part of the markdown. Also ensure that the stop-words you used are visible (either as image or as a vector).
Overall, I'd say no more than 2 hours of work, provided you diligently replicated the class work examples prior to this.
========
Task 2 – Basic NLP and Named Entity Extraction from one document.
Step 1 – Select one well-known firm from the list of the Fortune 500 firms.
Step 2 – For the selected firm, scrape it’s Wikipedia page.
Step 3 – Using openNLP, find all the locations and persons mentioned in the Wikipedia page. Its good practice to set timers and report runtimes for heavy functions.
Step 4 – Plot all the extracted locations from the Wikipedia page on a map. You may want to see 'NLP location extract and plot.R' file on LMS for this.
Step 5 – Briefly describe your observations based on the map and persons extracted from Wikipedia page.
Step 6 - Deliverable: R markdown submitted as HTML page. Ensure that the lists of people and persons mentioned are clearly visible. Also that the map of places is visible too. Use the text part of the markdown to record your observations.
Overall, I'd say no more than 1.5 hours of work.
Pls feel free to discuss these assignments with peers, help out and take help where necessary but at the end of the day, your submission must be individual and driven primarily by your own effort.
Any Qs etc, contact me, Vivek or Aashish.
Sudhir
This is a proforma welcome message for Batch 7 folks to the Text-Analytics for Business Applications (TABA) course.
Re what we did in the course:
We've gone through 5 TABA sessions which covered, respectively:
Session 1: Elementary Text-An + sentiment-An
Session 2: Data factorization + Topic Modeling 1
Session 3: Topic Modeling 2
Session 4: Basic NLP + PoS tagging + Named entity recognition (NER) and finally,
Session 5: Supervised text classification (using mainly the Maxent algo in R).
Re Readings and other material:
I shall put up links to relevant readings etc. In a lot of cases, just googling for particular topics can throw up a wealth of information.
The recommended textbook is theory-heavy and fairly comprehensive. I've linked a review of the same.
If you don;t have a free Github account, now is the time to get one and familiarize yourself with what Git is, how it works etc. For instance, here's a nice 10 minute read that can act as a starting guide.
I want you to think about sourcing some of your code, data etc directly from Git into Rstudio. I'll ask Pandey ji to address any of your queries on this score.
Re Home-works:
I'm committed to have your home-works (HWs) out early and on time. There'll be 2 individual and 2 group HWs.
The individual HWs will be simple, practice-based and clear-cut. Deliverable will be in R markdown form. I don;t expect any higher than 1.5-2 hrs of effort into each indiv HW.
The group HWs will be a little more comprehensive. Again, these too aren;t meant to be overly intricate or complex. Kindly ensure you're not over-thinking, over-analyzing or over-complicating any group HW.
Update:
I'm combining the 2 individual HWs into one. Likewise, combining the 2 group HWs into 1. So, there'll only be two HWs in TABA - one individual and one group.
Any Qs, comments, feedback etc. pls let me know.
Sudhir
Your final group homework is coming up soon. It involves scraping google search results and post-processing them a little bit. We will learn about text-processing techniques in some detail during your visit 2.
The idea is to scrape the top few pages of google search results for some of the world's most valuable brands and/or firms (both B2B and B2C)
And thereafter classify the URLs as originating from the firm or brand versus those originating from the rest of the world.
After that, a lot of interesting questions that can be asked and answered downstream.
Materials:
I'll be putting up the following materials up on linkstreet by this weekend.
Task:
1. I have assigned 8-10 brands to each group. Check the excel sheet on linkstreet for which brands have been assigned to your group. 2. Carefully go through the explanation in the word file for how to run the python script for scraping the data.
3. For each brand or firm in your list, run the data collection python script and collect data on the first 5 pages worth of search results. Use the keyword given as is in the search term.
4. Use the excel template to identify and mark each URL in the results as either "firm-sponsored" (if the link leads to a firm/brand sponsored page) or as "non-firm-sponsored" (if the contents of a URL are outside the brand's control. If neither holds, mark the link as "Irrelevant".
Deliverables:
Pls submit the excel sheets in the correct template for each brand (use separate worksheets in the same file for each brand). Name the excel sheet as "group HW 3 for group
Deadline:
Submit anytime before the midnight before (i) either the DC exam, or (ii) visit 2, whichever is earlier.
Any queries etc, contact me via the blog comments section or write to Vivek directly.
P.S. The results you bring in may be used for academic research. I'm hoping the results will be interesting enough to be publishable.
Sudhir
Pls find uploaded in linkstreet a markdown document for using the rvest package to scrape IMDB results.
Run the code line-by-line and check results. The code is documented to some extent. However, if you're unclear about a particular line is doing, web-search first, ask your peers, etc.
You should get a file containing URLs for the top 250 movies on IMDB. Once this is done, your real homework task begins.
Task:
1. Find all movies that released between 1996 and 1998 (both years including).
2. Read in the URL's imdb page and scrape the following information:
Director, stars, Taglines, Genres, (partial) storyline, Box office budget and box office gross.
3. Make a dataframe out of these variables as columns with movie name being the first variable.
4. Make a table movie-count versus Genres.
4a. Bonus points: See if you can come up with some interesting hypotheses. For example, you could hypothesize that "Action Genres occur significantly more often than Drama in the top-250 list." Or that "Action movies gross higher than Romance movies in the top 250 list." Etc.
5. Write a markdown doc with your code and explanation. See if you can storify your hypotheses.
Update: You're not restricted to only use rvest in R. If you're more comfortable with other platforms (say, Python 3), pls feel free to use the same to get the job done. But insert a markdown with the python code you used as part of your submission.
Deliverables and Deadline:
1. Zip a folder containing
2. Deliver the zipped folder to the appropriate dropbox.
3. Deadline will be the midnight of 23-October-2016.
Any Qs etc contact me. Or use the comments section below.
Sudhir
Wide range of topics we'd seen in the DC course. Some of you asked for more sources and reading material. Pls find the same below (in no particular order).
I do realize I got delayed in putting them out and some of you had followed up with me regarding this. Some readings are mandatory and others are totally optional only.
+++
The readings below are NOT optional in the sense that questions based on these readings may feature in your exam.
Readings relating technology to data collection and data use (from the Economist):
1. The first article titled 'Little Brother' (in an obvious play on George Orwell's famous 'big Brother' theme) details the impact of digital on advertising spends of firms worldwide.
2. The second article, 'Getting to know you', is about the various ways in which data is collected about consumers online.
3. The third article in this series, 'The world wild web', extrapolates some of what we are seeing into the future and asks 'Where are we going?'.
Ideally, I'd like you to read and discuss these articles within your groups. Again, remember, questions based on ideas and facts in these articles are fair game in your final exam for DC. Happy reading.
+++
Now, these readings that follow below are optional, more for leisure reading and folks with interest in particular topics/ verticals etc.
a. More from the Atlantic on how its now technologically feasible to arrive at one's Identity. Big Data Can Guess Who You Are Based on Your Zip Code.
b. Recall the habit patterns class we'd covered? Here's an article from HBR blogs on How Customers Get Hooked on Products.
c. There's an undercurrent somewhere in the program that spells the words "data science". This link here offers a rounded perspective on what precisely is data science. This follow-on link here describes 8 concrete steps you must take to become a data scientist. Yes, R features there. Apt read for all CBA students, IMO.
d. For sessions 1-3 which focussed more on constructs, designing questionnaires around constructs etc., here below is some interesting material which you may consider browsing at leisure. They're basically to help understanding for those folks who may have felt the coverage in class was not detailed enough on certain topics:
i. This is a Wikipedia link to Quantitative psychology as a subject area. It provides a nice, concise and precise introduction to the area in general and has a good number of downstream links that you can pick up on as and when necessary.
ii. This is the Wiki entry to Scaling techniques in general in the social sciences. As you can see the comparative versus noncomparative dichotomy comes in early on here. More links to detaiuled topics are also available.
iii. This is the wiki entry to psychometrics as a discipline. I thought it a tad too inclined towards educational testing but still, worth a read perhaps, for those interested.
+++
More reading links (optional) that I could gather, below. Recall the Xerox-evolv caselet we did in class - The one where psychographic Likerts were combined with straightforward machine learning? We then discussed implications, pros and cons etc, and speculated on when such practices may spread worldwide and into India. Well, speaking of India...
Startups, and India Inc use psychometric tests to peek into potential recruits’ minds
This is the session 5 reading from NYT on 'Data janitor' work: For Big-Data Scientists, ‘Janitor Work’ Is Key Hurdle to Insights
This is the source for the session 5 table on Data munging functions in R from computer world: Great R packages for data import, wrangling & visualization
This is the NYT article on 'the data driven life' about the quantified self movement.
This is the API directory from Programmable web. Recommended that as CBA folks, you register with the site and get periodic updates (if you haven't already). +++
You have 2 more group homeworks coming - one on webscraping using the rvest package and the other on python for scraping google search results. I'm teaching PGP currently, so posting a tad slow, but will put these HWs out there soon.
Any Qs etc, let me know.
Sudhir
I trust you've been busy with all the homework assignments etc from visit 1 visiting your weekends at home.
Your second individual homework is coming up. And not to worry, it won't be very burdensome, let me assure you. Please take this 20-30 minute web-survey. I plan to use this data to demonstrate concepts and techniques in the Text Analysis course during your next visit.
The survey should be completed (and is automatically submitted online) no later than the midnight of Sunday 16-Oct-2016 (i.e., a week from now).
I estimate the survey should take no more than 20-30 minutes.
+++
On a separate note, I recommend you register (free) with and subscribe to newsletters from the following sites:
1. www.kaggle.com which hosts data science competitions, releases datasets and tutorials. Can use your FB or goog login. We'll be using some kaggle datasets in NLP.
2. r-bloggers.com - which collates and publishes lots of interesting blogs on R, R news, new packages, cool coding hacks and tricks, etc.
3. 'programmable web' - which is sort of API central. Get news on updates, releases, changes etc in different popular APIs from all around.
Apart from the readings in past posts in this blog, this recent bloomberg article is of some interest from the DC and data science perspective.
Any Qs etc, pls feel free to contact me.
Sudhir
Vivek will upload all classwork related files - slides + code + data - in the next few days. Below are some tips for what to do re code and data
About Code:
About Data:
Pls ensure you are very comfortable with replicating classwork examples over the next week-10 days odd. The homeworks + deadlines will start coming in after that.
Recall that your assignment submissions will be in markdown format and the core insights therein should ideally should be wrapped in a narrative / storyline.
I've putup on linkstreet a file, RMarkdown.Rmd, that walks you through a simple procedure of how to write in markdown. The same I published on RPubs (and you can publish your markdowns too). It is available here. Here's a link to how markdown works. Its a very simple 1 page get-started guide in case you're 100% new to Markdowns.
And this page here, from the same author is about a few useful writing tools for how to craft a narrative into your markdowns. Again, use your judgment and don't follow the articles to the letter.
And this is an example of a markdown doc done for a project by somebody in 2014, published on Rpubs. Recall the blog page of Astronomer Julia Silge I'd shown in class? This is the link to her page. Its a nice intro to how to craft simple narratives around regular R code and workflows. Would be great if you can reproduce her workflow on your twitter feed, for instance...
OK. That's it for now. Feel free to use the comments section in case of anything.
Ciao and Cheers.
Sudhir
This group homework is based on (a) Session 1 - problem formulation, (b) Session 2 - survey questionnaire design and part of (c) Session 3 - Qualitative methods.
It aims to familiarize the intricacies, advantages and limitations of the Questionnaire tool by actively getting you to design that tool.
This is a group based homework. Only one submission per group. If you don't know who your group is, pls ask Vivek about this.
Problem Context
You are a startup bridging doctors and patients in metropolitan India. But your startup offers a house-call solution to elderly or otherwise infirm patients in the reasonably wealthy upper-middle class.
You connect doctors who are willing to make house calls (at some premium, presumably) with patients who are willing to pay the premium for the convenience of a house-call visit by a trustworthy doctor.
Imagine you are still in the process of figuring out the contours of their revenue/business model, which crucially depends what the demand levels are like for such services at what price, and what the supply levels are like at that price. Based on that one can figure out who to target, how and how large the market is.
Task
Your task is to (1) Formulate your D.P. and corresponding R.O.s. (My suggestion: Choose a sharply defined D.P. that can be well-covered by at most 1-2 R.O.s; choose a suitably narrow demographic as your target population that isn't too heterogeneous in its likely needs and preferences).
(2) Next, identify the main construct(s) of interest ones that will likely drive demand for your client, ideally based on your R.O.s. (Hint: Think in terms of the average target customer's motives/needs and the his/her self-perception of the requirements)
(3) Design a questionnaire that: [a] surveys target segment respondents on their propensity to use app-based on-demand commuting solution services;
[b] can be taken in under 15 minutes on a good net connection;
[c] collects info on the distribution of quantities of interest (such as awareness levels about these services, interest levels in using them, what price levels might be viable, etc.)
My suggestions before starting:
1. To understand this target segment's needs and preferences, do some preliminary, quick qualitative research: E.g., conduct a few interviews (these could be casual conversations or telephonic ones) with a few people in that target segment about the subject. Find out what they think, what they need, what they see others around them doing etc.
2. Write (or 'Program') your survey into either Qualtrics or the free versions of websurvey softwares available such as zoomerang or surveymonkey. See if you can obtain the "launch" survey link.
3. Bonus points for pretesting the survey with a few folks first, accounting for order effects etc in questionnaire design, etc.
Submission Format
You can submit a traditional PPT for this assignment. All future assignments will involve code and hence only markdown will be allowed.
Below is the submission format for a PPT. (That for a markdown will be similar. Just divide the markdown webpage into sections [Title, Data, model, results etc])
Start with a plain white PPT. Save it as groupName_session1.pptx
Title slide: Homework name and names+ roll numbers of group members.
First slide: Brief description of your client and their business (Also, a line or two to justify why you picked them)
Second slide: Statement of D.P and corresponding R.O.s
Third slide: Brief Description of qualitative research carried out to first narrow-down what topics to cover in the survey.
Fourth slide: Make a table that maps the 1-2 major constructs to the corresponding survey question numbers.
Fifth slide: Deliverable - websurvey link. Should be a working link. Also, attach the word or PDF version of your questionnaire onto this slide. The Q numbers in the 4th slide should match the ones here.
Sixth slide: Any learnings you as a group made - E.g., what constructs were the easiest to measure? hardest? ON the average, how many Qs per construct did you have to use? Etc.
Update:
In the past, I got quite a few Qs asking if a scale other than Likert can be used etc. Sure, it can. Likert is important in the context of behavioral constructs. For regular, descriptive Qs, use other scales by all means. *Not* every Q has to be a likert.
Whether PPT or markdown, wrap your submission inside a story, as far as possible.
Deadline for this is midnight of 16-Oct (Sunday). Drop box for the PPT will be set accordingly on linkstreet.
Any queries etc, let Vivek or me know.
Sudhir
A series of homeworks are coming your way. The first is described below. I will putup details for the others shortly.
Pls watch this ~ 20 minute video carefully. It features Scott McDonald of Condé Nast holding fort on where Marketing Research is headed in the next decade.
“Social Technological and Economic forces affecting Marketing Research over the next decade”
Now, for your HW, pls answer a few simple Qs (True-False, fill in the blanks variety) about the above talk in the following survey:
HW Notes:
(i) This is an individual-only HW. Since it involves no R, consulting peers is not permitted.
(ii) I found that using earphones works great in making out what the speaker is saying much more clearly than ordinary speakers. FYI.
(iii) Deadline: The HW should be completed and submitted latest by midnight 10-October Monday.
Any Qs etc, pls feel free to email me or use the comments section below.
Sudhir Voleti
