Category Archives: Big Data Analytics

Coffee with Ram Shriram

Ram Shriram (Shaerpalo Ventures profile) was in Bangalore for a day. He packed in 2 events at [24]/7’s new campus on Outer Ring Road. At the first event title In Conversation with Ram Shriram – Managing Partner, Sherpalo Ventures organised by iSpirt, he answered queries from scores of entrepreneurs. He wants more higher education institutions to come up, as it is tougher to get into IIT’s than Stanford. He cautioned against copy-cat mentality and app being the core of business, rather than actual revenue-based offering. You may get more details on Techinasia post – When the big daddy of startup investing Ram Shriram talks, you just listen!.

Tech stack at [24]/7

Tech stack at [24]/7

I was in the next event – Coffee with Ram Shriram. It was a more intimate setting. First, we were subjected to the mandatory and brief introduction about the company, its value proposition and strengths from a business guy named Brooks (who joined them post their Microsoft Tellme acquisition) and another head of technology. They talked about how [24]/7’s omni-channel approach for retails, backed by big data and prediction to provide a personalised and intuitive experience to the customers. We got a great demo of personalised experience based on web activity and intent. It showed how call center agents will be replaced by voice agents, leading the a lot of self-service by customers. They are coming up with exciting products in partnership with Facebook and Microsoft. If you want to know more about their work with Facebook Messenger, tune in to Facebook Messenger – The Most Disruptive Customer Support Channel Ever? on Dec 3. AS of Dec 23, you can hail Uber using Facebook Messenger in US.

Coffee with Ram Shriram at [24]/7

Coffee with Ram Shriram at [24]/7

Nags, co-founder of [24]/7 introduced Ram. Ram’s monologue was brief, where he touched upon technology evolution and future opportunities. This was followed by an interesting Q&A round. Responding to an audience question about he cherry-picks his angel investments like he did with Google he said – When he put in money, it was just 2 guys with a dream. He made sure that the dream was not short-lived looking at perseverance and talent of founders in addition to the market and incumbents. he used the same criteria, when he invested  in [24]/7. He provided clarity on [24]/7’s strategy and projections. [24]/7’s web, chat and voice offerings are disrupting the BPO business of incumbents, like Convergys as well as pure-product offerings, like Genesys, Oracle CRM etc. He also mentioned that they are starting to integrate with COTS (Commercial Off-the shelf) solutions like Dynamics CRM, etc soon for customers like Capital One. Varadh from [24]/7 then shared a thing or 2 about company culture, how they are applying the principles of delightful customer experience to an employee-friendly policies. He stopped short of asking us to check out the careers page of [24]/7’s Innovation Labs (direct link on Jobvite). We then continued the Q&A to the terrace with a great view of 10th floor.

ELK intro and Elasticsearch lessons from production

ELK stands for ElasticSearch, Logstash and Kibana. I had become acquainted with this during MongoDB Day – Bangalore on 19 May 2014 by Susheel Zaveri’s excellent talk. So, I was overjoyed, when the Elasticsearch Meetup Bangalore’s First Meetup coincided with my trip on 27 Sep 2014. Elasticsearch has got an open, RESTful API that makes it easy to build applications on top of it. It can process both structured and unstructured data, so you can derive insights from log files to Tweets to plain old CSV files, all in near real-time. Best of all, you can ingest data from all these disparate sources easily into Logstash, then search and analyze across all of these types of data with Elasticsearch, visualizing the results using Kibana. This stack makes these insights available to anyone in an organization through Kibana’s dashboards, which are share-able and don’t require programming know-how to use effectively.

These features – plus many more – make the ELK stack so flexible that it meets the big data challenges of a wide variety of verticals. A major financial company uses the ELK stack to do anomaly detection and root out credit card fraud. Another one performs analytics and sentiment analysis across social media data. Yet another one detects hacking on their networks, and yet another for full-text search across e-commerce sites with billions of entries.

Suyog Rao starts the talk while Drew sitsThe meetup was held at SpringPeople Software Pvt Ltd, Sector 7, HSR Layout, Bengaluru, Karnataka.  It had 2 speakers: Suyog Rao, Vedang Manerikar. It was free of cost, but required registration in a Google Form. Suyog Rao (@suyograo) started with an introduction to ELK. He started describing ElasticSearch as a schema-free, REST and JSON document store. The salient points of his talk were:

 

 

 

 

 

  • The popularity of ElasticSearch can be gauged from the total number of downloads, which stands at 10M in last 2 years.
  • An Elastic Search cluster can contain multiple Indices(databases), which in turn contain multiple Types(tables). These types hold multiple Documents (rows), and each document has Properties(columns). [Terms in bracket are relational counterpart]
  • It uses replication for high availability and performance. For horizontal scalability, it uses sharding.
  • It supports:
    • Unstructured as well as Faceted, structured search
    • Enrichment and sorting
    • Pagination and Aggregation

IMG_2731

He covered Logstash and Kibana next.

  • Logstash is a ruby app, which runs on JVM.
  • It allows one to collect, parse, enrich and store logs and events.
  • Kibana allows one to have beautiful visualization on top of Elasticsearch index with zero code.
  • The new version makes use D3 library.

He showed a quick demo. Actually covered a lot of stuff in short time.

 

 

 

Vedang Manerikar (@vedang) works with Helpshift, a mobile CRM company based out of Pune and San Francisco. [It’s a company, which has unique hiring practices. Refer my earlier blogpost on Building Silicon Valley culture in India]

IMG_2734The customer-facing side of Helpshift product is a simple chat feature within the app using the Helpshift mobile SDK. The business-facing side is a complex agent dashboard that helps the agent in processing as many issues as quickly as possible. This business-facing side is built on top of Elasticsearch. He shared the following nuggets of wisdom with us:

  • Elasticsearch does not have a book on it, although it will soon be solved. There are good references and videos, but nothing structured like a book yet.
  • Don’t use Elasticsearch as a primary database. The data should first go into mysql, MongoDB or other transactional datastore.
  • Though ES allows one to have a mixed mode node with both meta data and data, it is best to separate master and data nodes.
  • For multi-tenant index like Helpshift’s usecase, an index per customer is not a good idea, but something based on the index size.
  • He said helpful steps about bulk loading like controlling replica count etc, but I did not catch it fully.
  • Rolling upgrade of ES is fraught with risks, so it is better to spin up new cluster and decommission old one. [This was contested by Suyog and Drew]
  • Benchmarking is hugely important and should be done at staging and development phase to prevent aches during production. He mentioned about a tool called Tsung, which helped them benchmark percolators. Percolators allowed live notifications of new issues.
  • During runtime, a lot of debugging can be done using cat API’s, so make use of them.
  • Tune JVM parameters, like allocate more memory for young generation.
  • ES uses Lucene under the hood, so some troubleshooting might require understanding its working as well
  • RTFM – Basically read manual carefully. Pay special attention to the unit, whether a particular number refers to ms or seconds.
  • Advanced ES users make use of filters to make complex views.
  • There were many others, but I guess we have to wait for the presentation to arrive.

Bangalore top Indian city for professionals to move into among Indian cities

Linkedin had announced in June 2014, that it had a base of 26 million professionals in India, which is 2nd largest after USA – Hindu news article. Linkedin analyzed (original post on Linkedin blog) movement of technology professionals between Nov 2012 and Nov 2013. If we compare just the Indian cities, which appear in top 10 cities globally, Bangalore (or Bengaluru) comes to top in both absolute and percentage terms.

Linkedin Moving professionals

You may view the chart directly on Tableau public site.

Cloudera, an Apache Hadoop implementor raises an insane 900m

Cloudera just completed a $900M funding round. This incorporates the $160M they announced a week or so ago.

This is obviously great for Cloudera. For just 18% of the company they got $740M from Intel (plus sold away a few other equity points to other investors). It also signifies something more: The complete shape of data infrastructure is changing.

MongoDB has had a Hadoop connector for a while now. It has been used by several customers, like FourSquare. To cement this relationship more, Mike Olson, Cloudera co-founder and Chief Strategy Officer will keynote at MongoDB World this June at New York City.

[Disclaimer: I work as consulting engineer with MongoDB India. Shameless plug: If you want 10% discount for MongoDB World, please contact me prasoon DOT kumar AT mongodb DOT com]

%d bloggers like this: